Re: [R] multi-machine parallel setup?
On 21.04.2012 06:34, ivo welch wrote:
the vignette to the library(parallel) mentions snow repeatedly (esp
differences in its implementation in parallel from the original).
unfortunately, it doesn't give an example or tutorial for
multi-machine use with sockets.
could someone please point me to a simple working example, where a
master just has both itself and a 4-core slave on a second machine
(IP), and wants to execute an mclapply on a list on both machines,
like
master.ip- localhost
slave.ip- Rfriend@192.168.2.10 ## has key ssh access
mclapply( 1:100, function(x) { cat(x); Sys.sleep(1); x } ) # on both,
please
iaw
With appropriate ssh keys that do not need passwords and with the same
username on all machines, it should work out of the box. At least, it
does for me:
Say you want two, one on your own machine, the other one on your friends:
cl - makeCluster(c(localhost, 192.168.2.10), SOCK)
BEst,
Uwe Ligges
On Wed, Apr 18, 2012 at 1:01 PM, ivo welchivo.we...@gmail.com wrote:
Dear R experts:
could someone please point me to a page that explains how to set up
more than 1 machine for library parallel (which is quickly becoming my
favorite!)
...
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] multi-machine parallel setup?
hi uwe---my box is different:
cl - makeCluster(c(localhost, welch, calc), SOCK)
Error in loadNamespace(name) : there is no package called 'snow'
this seems to be part of snow, not parallel. ok, install snow first,
then rerun.
library(snow)
Attaching package: 'snow'
The following object(s) are masked from 'package:parallel':
clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
clusterExport, clusterMap, clusterSplit, makeCluster, parApply,
parCapply, parLapply,
parRapply, parSapply, splitIndices, stopCluster
bash: /Library/Frameworks/R.framework/Resources/bin/Rscript: No such
file or directory
but if I use two linux machines, it works.
now, how do I use parallel's mclapply with it?
best,
/iaw
Ivo Welch (ivo.we...@gmail.com)
J. Fred Weston Professor of Finance
Anderson School at UCLA, C519
http://www.ivo-welch.info/
Editor, Critical Finance Review, http://www.critical-finance-review.org/
2012/4/21 Uwe Ligges lig...@statistik.tu-dortmund.de:
On 21.04.2012 06:34, ivo welch wrote:
the vignette to the library(parallel) mentions snow repeatedly (esp
differences in its implementation in parallel from the original).
unfortunately, it doesn't give an example or tutorial for
multi-machine use with sockets.
could someone please point me to a simple working example, where a
master just has both itself and a 4-core slave on a second machine
(IP), and wants to execute an mclapply on a list on both machines,
like
master.ip- localhost
slave.ip- Rfriend@192.168.2.10 ## has key ssh access
mclapply( 1:100, function(x) { cat(x); Sys.sleep(1); x } ) # on both,
please
iaw
With appropriate ssh keys that do not need passwords and with the same
username on all machines, it should work out of the box. At least, it does
for me:
Say you want two, one on your own machine, the other one on your friends:
cl - makeCluster(c(localhost, 192.168.2.10), SOCK)
BEst,
Uwe Ligges
On Wed, Apr 18, 2012 at 1:01 PM, ivo welchivo.we...@gmail.com wrote:
Dear R experts:
could someone please point me to a page that explains how to set up
more than 1 machine for library parallel (which is quickly becoming my
favorite!)
...
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] multi-machine parallel setup?
On 21.04.2012 17:20, ivo welch wrote:
hi uwe---my box is different:
cl- makeCluster(c(localhost, welch, calc), SOCK)
Error in loadNamespace(name) : there is no package called 'snow'
this seems to be part of snow, not parallel. ok, install snow first,
then rerun.
No, you just need snow to be installed, using the functions from
parallel will load the snow Namespace, if installed.
library(snow)
Attaching package: 'snow'
The following object(s) are masked from 'package:parallel':
clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
clusterExport, clusterMap, clusterSplit, makeCluster, parApply,
parCapply, parLapply,
parRapply, parSapply, splitIndices, stopCluster
bash: /Library/Frameworks/R.framework/Resources/bin/Rscript: No such
file or directory
but if I use two linux machines, it works.
now, how do I use parallel's mclapply with it?
as I said.
As you seem to be under some heterogeneous setup (unstated before, as
far as I remember), you may need to specify paths to the loaction of R
etc. See the documentation for the parallel package.
Uwe
best,
/iaw
Ivo Welch (ivo.we...@gmail.com)
J. Fred Weston Professor of Finance
Anderson School at UCLA, C519
http://www.ivo-welch.info/
Editor, Critical Finance Review, http://www.critical-finance-review.org/
2012/4/21 Uwe Liggeslig...@statistik.tu-dortmund.de:
On 21.04.2012 06:34, ivo welch wrote:
the vignette to the library(parallel) mentions snow repeatedly (esp
differences in its implementation in parallel from the original).
unfortunately, it doesn't give an example or tutorial for
multi-machine use with sockets.
could someone please point me to a simple working example, where a
master just has both itself and a 4-core slave on a second machine
(IP), and wants to execute an mclapply on a list on both machines,
like
master.ip- localhost
slave.ip- Rfriend@192.168.2.10 ## has key ssh access
mclapply( 1:100, function(x) { cat(x); Sys.sleep(1); x } ) # on both,
please
iaw
With appropriate ssh keys that do not need passwords and with the same
username on all machines, it should work out of the box. At least, it does
for me:
Say you want two, one on your own machine, the other one on your friends:
cl- makeCluster(c(localhost, 192.168.2.10), SOCK)
BEst,
Uwe Ligges
On Wed, Apr 18, 2012 at 1:01 PM, ivo welchivo.we...@gmail.comwrote:
Dear R experts:
could someone please point me to a page that explains how to set up
more than 1 machine for library parallel (which is quickly becoming my
favorite!)
...
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] multi-machine parallel setup?
On Sat, Apr 21, 2012 at 8:20 AM, ivo welch ivo.we...@gmail.com wrote:
hi uwe---my box is different:
cl - makeCluster(c(localhost, welch, calc), SOCK)
Error in loadNamespace(name) : there is no package called 'snow'
this seems to be part of snow, not parallel. ok, install snow first,
then rerun.
library(snow)
Attaching package: 'snow'
The following object(s) are masked from 'package:parallel':
clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
clusterExport, clusterMap, clusterSplit, makeCluster, parApply,
parCapply, parLapply,
parRapply, parSapply, splitIndices, stopCluster
bash: /Library/Frameworks/R.framework/Resources/bin/Rscript: No such
file or directory
but if I use two linux machines, it works.
now, how do I use parallel's mclapply with it?
parLapply(cl = cl, X = data, fun = mean)
AFAIK, mclapply() does not take a cluster argument, but the par* functions do.
Cheers,
Josh
best,
/iaw
Ivo Welch (ivo.we...@gmail.com)
J. Fred Weston Professor of Finance
Anderson School at UCLA, C519
http://www.ivo-welch.info/
Editor, Critical Finance Review, http://www.critical-finance-review.org/
2012/4/21 Uwe Ligges lig...@statistik.tu-dortmund.de:
On 21.04.2012 06:34, ivo welch wrote:
the vignette to the library(parallel) mentions snow repeatedly (esp
differences in its implementation in parallel from the original).
unfortunately, it doesn't give an example or tutorial for
multi-machine use with sockets.
could someone please point me to a simple working example, where a
master just has both itself and a 4-core slave on a second machine
(IP), and wants to execute an mclapply on a list on both machines,
like
master.ip- localhost
slave.ip- Rfriend@192.168.2.10 ## has key ssh access
mclapply( 1:100, function(x) { cat(x); Sys.sleep(1); x } ) # on both,
please
iaw
With appropriate ssh keys that do not need passwords and with the same
username on all machines, it should work out of the box. At least, it does
for me:
Say you want two, one on your own machine, the other one on your friends:
cl - makeCluster(c(localhost, 192.168.2.10), SOCK)
BEst,
Uwe Ligges
On Wed, Apr 18, 2012 at 1:01 PM, ivo welchivo.we...@gmail.com wrote:
Dear R experts:
could someone please point me to a page that explains how to set up
more than 1 machine for library parallel (which is quickly becoming my
favorite!)
...
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
Joshua Wiley
Ph.D. Student, Health Psychology
Programmer Analyst II, Statistical Consulting Group
University of California, Los Angeles
https://joshuawiley.com/
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] multi-machine parallel setup?
thx, guys, almost there. This is good fodder for the vignette or ?parallel.
Steps:
(1) install package snow on all machines which you want to be part
of a cluster.
(2) run under R
library(parallel)
cl - makeCluster(c(localhost, calc.localdomain), SOCK)
result - parLapply(cl=cl, X=1:100, function(x) {
paste(x,system(hostname, intern=TRUE)) } )
stopCluster(cl)
in my case, with two linux ubuntu machines and R 2.14...this hangs (!)
because I made a small mistake, using TRUE instead of TRUE. if I
fix it, it works fine. is there a way to have the slave signal abort
the parLapply job when it encounters an error? [or is it something
with my setup that prevents the abort? does it abort correctly on
your machines?]
(I run not R 2.15 but R 2.14.1 on ubuntu 12.04LTS beta, because in
/etc/apt/sources.list, deb http://cran.cnr.Berkeley.edu precise
multiverse still installs 2.14.1.)
regards,
/iaw
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] multi-machine parallel setup?
the vignette to the library(parallel) mentions snow repeatedly (esp
differences in its implementation in parallel from the original).
unfortunately, it doesn't give an example or tutorial for
multi-machine use with sockets.
could someone please point me to a simple working example, where a
master just has both itself and a 4-core slave on a second machine
(IP), and wants to execute an mclapply on a list on both machines,
like
master.ip - localhost
slave.ip - Rfriend@192.168.2.10 ## has key ssh access
mclapply( 1:100, function(x) { cat(x); Sys.sleep(1); x } ) # on both, please
iaw
On Wed, Apr 18, 2012 at 1:01 PM, ivo welch ivo.we...@gmail.com wrote:
Dear R experts:
could someone please point me to a page that explains how to set up
more than 1 machine for library parallel (which is quickly becoming my
favorite!)
...
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
