Re: [R] parallel processing in r...
Hi,
On ' how to use "top" inside the R prompt? '
you can use system('top') command.
Thanks,
On Sun, Jul 1, 2018 at 9:53 PM Benoit Vaillant
wrote:
> Hello,
>
> On Sun, Jul 01, 2018 at 11:31:29AM +, akshay kulkarni wrote:
> > I tried "top" at the bash prompt, but it provides a way to measure
> > CPU performance of the existing processes. I want to check the CPU
> > usage of the execution of an R function.
>
> Try to open two bash prompts, in one use R and in the other use top to
> monitor what is going on.
>
> > and at the R prompt I type the function to be executed. But if I
> > type "top" at the R prompt, it says object "top" not found.
>
> top is a shell command, no issue with R not knowing about this.
>
> > So, should I change to bash prompt after running the R function? If
> > yes, how do I do it? If not, how to use "top" inside the R prompt?
>
> Basically, you can't.
>
> > Again, I think this is an OS isuuebut I could'nt find any answer
> > in the Internet. I am an independent researcher and I don't have
> > personal access to experts...this mail list is the only vent I
> > have...
>
> ... (many more dots) Do you think we are experts on your system?
>
> Please do your home work and get back to us once it's done. ;-)
>
> Cheers,
>
> --
> Benoît
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing in r...
Hello, On Sun, Jul 01, 2018 at 11:31:29AM +, akshay kulkarni wrote: > I tried "top" at the bash prompt, but it provides a way to measure > CPU performance of the existing processes. I want to check the CPU > usage of the execution of an R function. Try to open two bash prompts, in one use R and in the other use top to monitor what is going on. > and at the R prompt I type the function to be executed. But if I > type "top" at the R prompt, it says object "top" not found. top is a shell command, no issue with R not knowing about this. > So, should I change to bash prompt after running the R function? If > yes, how do I do it? If not, how to use "top" inside the R prompt? Basically, you can't. > Again, I think this is an OS isuuebut I could'nt find any answer > in the Internet. I am an independent researcher and I don't have > personal access to experts...this mail list is the only vent I > have... ... (many more dots) Do you think we are experts on your system? Please do your home work and get back to us once it's done. ;-) Cheers, -- Benoît signature.asc Description: PGP signature __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing in r...
dear Members, Thanks for the reply..I do have another issue; I will be highly obliged if you answer it: I tried "top" at the bash prompt, but it provides a way to measure CPU performance of the existing processes. I want to check the CPU usage of the execution of an R function. So I start R by this $ R and at the R prompt I type the function to be executed. But if I type "top" at the R prompt, it says object "top" not found. So, should I change to bash prompt after running the R function? If yes, how do I do it? If not, how to use "top" inside the R prompt? Again, I think this is an OS isuuebut I could'nt find any answer in the Internet. I am an independent researcher and I don't have personal access to experts...this mail list is the only vent I have... Very many thanks for your time and effort... Yours sincerely, AKSHAY M KULKARNI From: Jeff Newmiller Sent: Saturday, June 30, 2018 11:46 PM To: r-help@r-project.org; akshay kulkarni; R help Mailing list Subject: Re: [R] parallel processing in r... Use "top" at the bash prompt. Read about the "mc.cores" parameter to mclapply. Make a simplified example version of your analysis and post your question in the context of that example [1][2][3]. You will learn about the issues you are dealing with in the process of trimming your problem, and will have code you can share that demonstrates the issue without exposing private information. Running parallel does not necessarily improve performance because other factors like task switching overhead and Inter-process-communication (data sharing) can drag it down. Read about the real benefits and drawbacks of parallelism... there are many discussions out there out there... you might start with [4]. [1] http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example [2] http://adv-r.had.co.nz/Reproducibility.html [3] https://cran.r-project.org/web/packages/reprex/index.html (read the vignette) [4] https://nceas.github.io/oss-lessons/parallel-computing-in-r/parallel-computing-in-r.html On June 30, 2018 10:07:49 AM PDT, akshay kulkarni wrote: >dear members, >I am using mclapply to parallelize my code. I am using Red Hat Linux in >AWS. > >When I use mclapply, I see no speed increase. I doubt that the Linux OS >is allowing fewer than the maximum number of cores to mclapply ( by >default, mclapply takes all the available cores to it). > >How do you check if the number of workers is less than the output given >by detectCores(), in Linux? Is there any R function for it? > >I do acknowledge that help on an OS is not suitable for this mailing >list, but even Internet could'nt help me. Therefore this mail.. > >very many thanks for your time and effort... >yours sincerely, >AKSHAY M KULKARNI > > [[alternative HTML version deleted]] > >__ >R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code. -- Sent from my phone. Please excuse my brevity. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing in r...
If you use gkrellm, you'll get a plot of each core's activity so it's easy to see how many are being used. yum install gkrellm. HTH On 07/01/2018 06:16 AM, Jeff Newmiller wrote: > Use "top" at the bash prompt. > > Read about the "mc.cores" parameter to mclapply. > > Make a simplified example version of your analysis and post your question in > the context of that example [1][2][3]. You will learn about the issues you > are dealing with in the process of trimming your problem, and will have code > you can share that demonstrates the issue without exposing private > information. > > Running parallel does not necessarily improve performance because other > factors like task switching overhead and Inter-process-communication (data > sharing) can drag it down. Read about the real benefits and drawbacks of > parallelism... there are many discussions out there out there... you might > start with [4]. > > > [1] > http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example > > [2] http://adv-r.had.co.nz/Reproducibility.html > > [3] https://cran.r-project.org/web/packages/reprex/index.html (read the > vignette) > > [4] > https://nceas.github.io/oss-lessons/parallel-computing-in-r/parallel-computing-in-r.html > > On June 30, 2018 10:07:49 AM PDT, akshay kulkarni > wrote: >> dear members, >> I am using mclapply to parallelize my code. I am using Red Hat Linux in >> AWS. >> >> When I use mclapply, I see no speed increase. I doubt that the Linux OS >> is allowing fewer than the maximum number of cores to mclapply ( by >> default, mclapply takes all the available cores to it). >> >> How do you check if the number of workers is less than the output given >> by detectCores(), in Linux? Is there any R function for it? >> >> I do acknowledge that help on an OS is not suitable for this mailing >> list, but even Internet could'nt help me. Therefore this mail.. >> >> very many thanks for your time and effort... >> yours sincerely, >> AKSHAY M KULKARNI >> >> [[alternative HTML version deleted]] >> >> __ >> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing in r...
Use "top" at the bash prompt. Read about the "mc.cores" parameter to mclapply. Make a simplified example version of your analysis and post your question in the context of that example [1][2][3]. You will learn about the issues you are dealing with in the process of trimming your problem, and will have code you can share that demonstrates the issue without exposing private information. Running parallel does not necessarily improve performance because other factors like task switching overhead and Inter-process-communication (data sharing) can drag it down. Read about the real benefits and drawbacks of parallelism... there are many discussions out there out there... you might start with [4]. [1] http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example [2] http://adv-r.had.co.nz/Reproducibility.html [3] https://cran.r-project.org/web/packages/reprex/index.html (read the vignette) [4] https://nceas.github.io/oss-lessons/parallel-computing-in-r/parallel-computing-in-r.html On June 30, 2018 10:07:49 AM PDT, akshay kulkarni wrote: >dear members, >I am using mclapply to parallelize my code. I am using Red Hat Linux in >AWS. > >When I use mclapply, I see no speed increase. I doubt that the Linux OS >is allowing fewer than the maximum number of cores to mclapply ( by >default, mclapply takes all the available cores to it). > >How do you check if the number of workers is less than the output given >by detectCores(), in Linux? Is there any R function for it? > >I do acknowledge that help on an OS is not suitable for this mailing >list, but even Internet could'nt help me. Therefore this mail.. > >very many thanks for your time and effort... >yours sincerely, >AKSHAY M KULKARNI > > [[alternative HTML version deleted]] > >__ >R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code. -- Sent from my phone. Please excuse my brevity. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing in r...
The effectiveness of parallelizing code, be it with mclapply or otherwise, depends in large part on the code, which you failed to show. I cannot answer your other question. Cheers, Bert Bert Gunter "The trouble with having an open mind is that people keep coming along and sticking things into it." -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) On Sat, Jun 30, 2018 at 10:07 AM, akshay kulkarni wrote: > dear members, > I am using mclapply to parallelize my code. I > am using Red Hat Linux in AWS. > > When I use mclapply, I see no speed increase. I doubt that the Linux OS is > allowing fewer than the maximum number of cores to mclapply ( by default, > mclapply takes all the available cores to it). > > How do you check if the number of workers is less than the output given by > detectCores(), in Linux? Is there any R function for it? > > I do acknowledge that help on an OS is not suitable for this mailing list, > but even Internet could'nt help me. Therefore this mail.. > > very many thanks for your time and effort... > yours sincerely, > AKSHAY M KULKARNI > > [[alternative HTML version deleted]] > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/ > posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] parallel processing in r...
dear members, I am using mclapply to parallelize my code. I am using Red Hat Linux in AWS. When I use mclapply, I see no speed increase. I doubt that the Linux OS is allowing fewer than the maximum number of cores to mclapply ( by default, mclapply takes all the available cores to it). How do you check if the number of workers is less than the output given by detectCores(), in Linux? Is there any R function for it? I do acknowledge that help on an OS is not suitable for this mailing list, but even Internet could'nt help me. Therefore this mail.. very many thanks for your time and effort... yours sincerely, AKSHAY M KULKARNI [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Parallel Processing
Helllo All, Need some help understanding parallel processing. I set-up DoParallel and worked perfectly. I tried to set-up using parallel package following the book Parallel R but I get the following error: Error in checkForRemoteErrors(val) : 4 nodes produced errors; first error: 'what' must be a character string or a function I have read some posts on the internet suggesting the issue may be related to S4 class and I have tried parse() to no avail. I have read Snow and Parallel documentation but I am not getting the issue. Glenn library(parallel) library(foreach) library(doParallel) From Parallel R and Internet set up the following DoParallel 4 seconds (perfect) took about 10 mins to set-up == ptm - proc.time() Cluster - makeCluster(detectCores()) registerDoParallel(Cluster) clusterSetRNGStream(cl = Cluster, set.seed(100)) OAS - foreach(i = 1:4, .packages = c(BondLab), .combine = cbind) %dopar% Mortgage.OAS(bond.id = bondlabMBS4, trade.date = 01-10-2013, settlement.date = 01-13-2013, original.bal = 10, price = 105.75, sigma = .23/sqrt(240), paths = 50)@OAS mean(OAS) proc.time() - ptm hist(OAS * 100, breaks = 20) stopCluster(Cluster) == followed the book example fails with the following message: Error in checkForRemoteErrors(val) : 4 nodes produced errors; first error: 'what' must be a character string or a function ptm - proc.time() RNGkind(L'Ecuyer-CMRG) cl - makeCluster(detectCores()) clusterSetRNGStream(cl, set.seed(100)) clusterEvalQ(cl, library(BondLab)) OAS - clusterApply(cl = cl, 1:4, Mortgage.OAS(bond.id = bondlabMBS4, trade.date = 01-10-2013, settlement.date = 01-13-2013, original.bal = 10, price = 105.75, sigma = .23/sqrt(240), paths = 50)@OAS) proc.time() - ptm mean(OAS) hist(OAS * 100, breaks = 20) stopCluster(cl) __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Parallel processing
Dear community, Sory for cross posting. Does anybody have an idea on how I can do parallel in MATLAB? thanks for your help -- John Wasige [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing in Matlab (irrelevant to R)
Sorry, I think you posted to the wrong group. Ranjan On Sat, 11 Apr 2015 19:01:04 +0200 John Wasige johnwas...@gmail.com wrote: Dear community, Sory for cross posting. Does anybody have an idea on how I can do parallel in MATLAB? thanks for your help -- John Wasige [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Important Notice: This mailbox is ignored: e-mails are set to be deleted on receipt. Please respond to the mailing list if appropriate. For those needing to send personal or professional e-mail, please use appropriate addresses. FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop! __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing
Wrong mailinglist. This one is about R, not matlab. Op 11-apr.-2015 19:03 schreef John Wasige johnwas...@gmail.com: Dear community, Sory for cross posting. Does anybody have an idea on how I can do parallel in MATLAB? thanks for your help -- John Wasige [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Parallel processing random 'save' error
Platform: Windows 7
Package: parallel
Function: parLapply
I am running a lengthy program with 8 parallel processes running in main
memory.
The processes save data using the 'save' function, to distinct files so
that no conflicts writing to the same file are possible.
I have been getting errors like the one shown below on a random basis,
i.e., sometimes at one point in the execution, sometimes at another,
sometimes no error at all.
I should note that the directory referred to in the error message
( 'D:\_pgf\quantile_analysis2_f13\_save\dbz084_nump48\bins') contains, as I
write, 124 files saved to it by the program without any error; which
underscores the point that most of the time the saves occur with no problem.
Error in checkForRemoteErrors(val) :
one node produced an error: (converted from warning)
'D:\_pgf\quantile_analysis2_f13\_save\dbz084_nump48\bins' already exists
Enter a frame number, or 0 to exit
1: main_top(9)
2: main_top.r#26: eval(call_me)
3: eval(expr, envir, enclos)
4: quantile_analysis(2)
5: quantile_analysis.r#69: run_all(layr, prjp, np, rules_tb, pctiles_tb,
parx, logdir, logg)
6: run_all.r#73: parLapply(cl, ctrl_all$vn, qa1, prjp, dfr1, iu__bool,
parx, logdir, tstamp)
7: do.call(c, clusterApply(cl, x = splitList(X, length(cl)), fun = lapply,
fun, ...), quote = TRUE)
8: clusterApply(cl, x = splitList(X, length(cl)), fun = lapply, fun, ...)
9: staticClusterApply(cl, fun, length(x), argfun)
10: checkForRemoteErrors(val)
11: stop(one node produced an error: , firstmsg, domain = NA)
12: (function ()
{
error()
utils::recover()
})()
Following the latest error I checked the system's connections as follows:
Browse[1] showConnections()
description class mode text isopen can read can
write
3 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
4 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
5 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
6 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
7 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
8 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
9 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
10 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
Browse[1]
It seems that the parallel processes might be sharing the same
connection--or is it that they are utilizing connections that have the same
name but are actually distinct because they are running in parallel?
If the connections are the problem, how can I force each parallel process
to use a different connection?
If the connections are not the problem, then can someone suggest a
diagnostic I might apply to tease out what is going wrong? Or perhaps some
program setting that I may have neglected to consider?
Thanks in advance for your help.
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing random 'save' error
Error in checkForRemoteErrors(val) :
one node produced an error: (converted from warning)
'D:\_pgf\quantile_analysis2_f13\_save\dbz084_nump48\bins' already exists
That warning looks like it comes from dir.create(). Do you have
code that looks like:
if (!file.exists(tempDir)) {
dir.create(tempDir)
}
If so that could be the problem. The directory may not exist when file.exists()
is called but by the time dir.create is called another process may have created
it. Try replacing such code with
suppressWarnings(dir.create(tempDir))
if (!isTRUE(file.info(tempDir)$isdir)) {
stop(Cannot create tempDir=, tempDir)
}
Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com
-Original Message-
From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On
Behalf
Of Rguy
Sent: Monday, July 01, 2013 1:07 AM
To: r-help@r-project.org
Subject: [R] Parallel processing random 'save' error
Platform: Windows 7
Package: parallel
Function: parLapply
I am running a lengthy program with 8 parallel processes running in main
memory.
The processes save data using the 'save' function, to distinct files so
that no conflicts writing to the same file are possible.
I have been getting errors like the one shown below on a random basis,
i.e., sometimes at one point in the execution, sometimes at another,
sometimes no error at all.
I should note that the directory referred to in the error message
( 'D:\_pgf\quantile_analysis2_f13\_save\dbz084_nump48\bins') contains, as I
write, 124 files saved to it by the program without any error; which
underscores the point that most of the time the saves occur with no problem.
Error in checkForRemoteErrors(val) :
one node produced an error: (converted from warning)
'D:\_pgf\quantile_analysis2_f13\_save\dbz084_nump48\bins' already exists
Enter a frame number, or 0 to exit
1: main_top(9)
2: main_top.r#26: eval(call_me)
3: eval(expr, envir, enclos)
4: quantile_analysis(2)
5: quantile_analysis.r#69: run_all(layr, prjp, np, rules_tb, pctiles_tb,
parx, logdir, logg)
6: run_all.r#73: parLapply(cl, ctrl_all$vn, qa1, prjp, dfr1, iu__bool,
parx, logdir, tstamp)
7: do.call(c, clusterApply(cl, x = splitList(X, length(cl)), fun = lapply,
fun, ...), quote = TRUE)
8: clusterApply(cl, x = splitList(X, length(cl)), fun = lapply, fun, ...)
9: staticClusterApply(cl, fun, length(x), argfun)
10: checkForRemoteErrors(val)
11: stop(one node produced an error: , firstmsg, domain = NA)
12: (function ()
{
error()
utils::recover()
})()
Following the latest error I checked the system's connections as follows:
Browse[1] showConnections()
description class mode text isopen can read can
write
3 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
4 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
5 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
6 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
7 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
8 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
9 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
10 -LAPTOP_32G_01:11741 sockconn a+b binary opened yes
yes
Browse[1]
It seems that the parallel processes might be sharing the same
connection--or is it that they are utilizing connections that have the same
name but are actually distinct because they are running in parallel?
If the connections are the problem, how can I force each parallel process
to use a different connection?
If the connections are not the problem, then can someone suggest a
diagnostic I might apply to tease out what is going wrong? Or perhaps some
program setting that I may have neglected to consider?
Thanks in advance for your help.
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] parallel processing with foreach
Hi,
I am trying to parallel computing with foreach function, but not able to
get the result. I know that in parallel processing, all result is collected
in list format, but I am not able to get input there.
Any help is really appreciated.
esf.m -foreach (i = 1:n.s, .combine=rbind) %dopar% {
EV - as.data.frame(eig$vectors[,1:n.candid[i]])
colnames(EV) - paste(EV, 1:NCOL(EV), sep=)
r25.esf.f - lm(y ~ x1 + x2 +., data = EV)
assign(paste(r25.esf., i, sep=), stepwise.forward(r25.esf.f, lm(y ~ x1
+ x2, data = EV), 0.1, verbose = F))}
--
View this message in context:
http://r.789695.n4.nabble.com/parallel-processing-with-foreach-tp4647381.html
Sent from the R help mailing list archive at Nabble.com.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing with foreach
It seems you don't quite understand how foreach works. foreach (..)
%dopar% { ... } takes the last value from each of the second {...}
evaluations and feeds them to the .combine function (in your case
rbind()). Since your last call in the %dopar% {...} block is assign(),
you are not getting anything meaningful.
Make the last value a vector that you want to be rbind-ed to the result.
HTH,
Peter
On Thu, Oct 25, 2012 at 1:47 AM, pxs101220 pxs101...@utdallas.edu wrote:
Hi,
I am trying to parallel computing with foreach function, but not able to
get the result. I know that in parallel processing, all result is collected
in list format, but I am not able to get input there.
Any help is really appreciated.
esf.m -foreach (i = 1:n.s, .combine=rbind) %dopar% {
EV - as.data.frame(eig$vectors[,1:n.candid[i]])
colnames(EV) - paste(EV, 1:NCOL(EV), sep=)
r25.esf.f - lm(y ~ x1 + x2 +., data = EV)
assign(paste(r25.esf., i, sep=), stepwise.forward(r25.esf.f, lm(y ~ x1
+ x2, data = EV), 0.1, verbose = F))}
--
View this message in context:
http://r.789695.n4.nabble.com/parallel-processing-with-foreach-tp4647381.html
Sent from the R help mailing list archive at Nabble.com.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing with multiple directories
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 17/04/12 20:06, David Schaefer wrote:
Hello,
I would like to run some code in parallel with each cluster reading/writing
to a different
working directory. I've tried the following code without success. The error
I get is: Error
in setwd(x) : cannot change working directory
1) more likely to get a response from the sig-hpc mailing list as here
2) What scheduler is the cluster using? Some use the current working directory,
others change the
working directory - so check there
3) What I did in a similar situation (cluster using torque and maoi) was to use
tempdir to creat
the directory in the script - might work for you or not
4) to use print(getwd()) in your script is a good idea - but you could even use
echo `pwd`
in your submit script to check
Many possibilities what is going wrong, many solutions - we need more info
Cheers,
Rainer
library(parallel) dirs - list(out1,out2,out3) # these directories
are located within
the current working directory temp - 1:3 testF - function(x) { setwd(x)
saveRDS(temp,temp.drs) } mclapply(dirs, testF)
Any help would be appreciated!
--David
* David R. Schaefer, Ph.D.
Assistant Professor
School of Social and Family Dynamics Arizona State University
www.public.asu.edu/~schaef/
__ R-help@r-project.org mailing
list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html and provide commented, minimal,
self-contained,
reproducible code.
- --
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology,
UCT), Dipl. Phys.
(Germany)
Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa
Tel : +33 - (0)9 53 10 27 44
Cell: +33 - (0)6 85 62 59 98
Fax : +33 - (0)9 58 10 27 44
Fax (D):+49 - (0)3 21 21 25 22 44
email: rai...@krugs.de
Skype: RMkrug
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk+OaZEACgkQoYgNqgF2egpY6wCfaPNnuUIrOiHv+T6G8mnmxgIJ
4NQAn0S+vmTP06jkHVJAayBcoCNYoqav
=RqJW
-END PGP SIGNATURE-
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing with multiple directories
Hi Rainier,
Thanks for your suggestions. I should have been more specific, I am using
multiple cores on a Mac Pro running Snow Leopard. I can see where that makes a
difference.
--David
On 4/18/12 12:13 AM, Rainer M Krug r.m.k...@gmail.com wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 17/04/12 20:06, David Schaefer wrote:
Hello,
I would like to run some code in parallel with each cluster reading/writing
to a different
working directory. I've tried the following code without success. The error
I get is: Error
in setwd(x) : cannot change working directory
1) more likely to get a response from the sig-hpc mailing list as here
2) What scheduler is the cluster using? Some use the current working directory,
others change the
working directory - so check there
3) What I did in a similar situation (cluster using torque and maoi) was to use
tempdir to creat
the directory in the script - might work for you or not
4) to use print(getwd()) in your script is a good idea - but you could even use
echo `pwd`
in your submit script to check
Many possibilities what is going wrong, many solutions - we need more info
Cheers,
Rainer
library(parallel) dirs - list(out1,out2,out3) # these directories
are located within
the current working directory temp - 1:3 testF - function(x) { setwd(x)
saveRDS(temp,temp.drs) } mclapply(dirs, testF)
Any help would be appreciated!
--David
* David R. Schaefer, Ph.D.
Assistant Professor
School of Social and Family Dynamics Arizona State University
www.public.asu.edu/~schaef/
__ R-help@r-project.org mailing
list
https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html and provide commented, minimal,
self-contained,
reproducible code.
- --
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology,
UCT), Dipl. Phys.
(Germany)
Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa
Tel : +33 - (0)9 53 10 27 44
Cell: +33 - (0)6 85 62 59 98
Fax : +33 - (0)9 58 10 27 44
Fax (D):+49 - (0)3 21 21 25 22 44
email: rai...@krugs.de
Skype: RMkrug
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk+OaZEACgkQoYgNqgF2egpY6wCfaPNnuUIrOiHv+T6G8mnmxgIJ
4NQAn0S+vmTP06jkHVJAayBcoCNYoqav
=RqJW
-END PGP SIGNATURE-
*
David R. Schaefer, Ph.D.
Assistant Professor
School of Social and Family Dynamics
Arizona State University
www.public.asu.edu/~schaef/
*
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] parallel processing with multiple directories
Hello,
I would like to run some code in parallel with each cluster reading/writing to
a different working directory. I've tried the following code without success.
The error I get is: Error in setwd(x) : cannot change working directory
library(parallel)
dirs - list(out1,out2,out3) # these directories are located within the
current working directory
temp - 1:3
testF - function(x) {
setwd(x)
saveRDS(temp,temp.drs)
}
mclapply(dirs, testF)
Any help would be appreciated!
--David
*
David R. Schaefer, Ph.D.
Assistant Professor
School of Social and Family Dynamics
Arizona State University
www.public.asu.edu/~schaef/
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing with multiple directories
On Tue, Apr 17, 2012 at 11:06:05AM -0700, David Schaefer wrote:
Hello,
I would like to run some code in parallel with each cluster reading/writing
to a different working directory. I've tried the following code without
success. The error I get is: Error in setwd(x) : cannot change working
directory
library(parallel)
dirs - list(out1,out2,out3) # these directories are located within
the current working directory
temp - 1:3
testF - function(x) {
setwd(x)
saveRDS(temp,temp.drs)
}
mclapply(dirs, testF)
Hi.
Try to include
print(getwd())
to testF() function, so that you see, where the script is running.
Another option is not to use setwd() and use
saveRDS(temp, paste(x, temp.drs, sep=/))
instead of saveRDS(temp,temp.drs).
Hope this helps.
Petr Savicky.
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] parallel processing with multiple directories
On 04/17/2012 11:06 AM, David Schaefer wrote:
Hello,
I would like to run some code in parallel with each cluster reading/writing to a
different working directory. I've tried the following code without success. The error I
get is: Error in setwd(x) : cannot change working directory
library(parallel)
dirs- list(out1,out2,out3) # these directories are located within the
current working directory
temp- 1:3
testF- function(x) {
setwd(x)
saveRDS(temp,temp.drs)
}
mclapply(dirs, testF)
Any help would be appreciated!
Hi David,
Suppose that jobs 1 and 3 are executed on processor 2. Then after the
first iteration the directory is ./out1 and on the second iteration
setwd() tries to change to ./out1/out3. Full path names might help.
My unasked-for advice would be to put 'more' of the processing in testF,
so that it is reasonable to return and then aggregate the results
without writing to disk.
Martin
--David
*
David R. Schaefer, Ph.D.
Assistant Professor
School of Social and Family Dynamics
Arizona State University
www.public.asu.edu/~schaef/
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
Computational Biology
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109
Location: M1-B861
Telephone: 206 667-2793
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing for R loop
Hi Sandeep, still missing an answer? Perhaps you cross check your post with the rules of the posting guide and find what is missing at all here. Anyway, depending on your OS, package multicore, snow/snowfall may fit your needs - but you have to re-formulate your loop using adequate multicore *apply-functions. hth Am 11.10.2011 14:13, schrieb Sandeep Patil: I have an R script that consists of a for loop that repeats a process for many different files. I want to process this parallely on machine with multiple cores, is there any package for it ? Thanks -- Eik Vettorazzi Department of Medical Biometry and Epidemiology University Medical Center Hamburg-Eppendorf Martinistr. 52 20246 Hamburg T ++49/40/7410-58243 F ++49/40/7410-57790 -- Pflichtangaben gemäß Gesetz über elektronische Handelsregister und Genossenschaftsregister sowie das Unternehmensregister (EHUG): Universitätsklinikum Hamburg-Eppendorf; Körperschaft des öffentlichen Rechts; Gerichtsstand: Hamburg Vorstandsmitglieder: Prof. Dr. Guido Sauter (Vertreter des Vorsitzenden), Dr. Alexander Kirstein, Joachim Prölß, Prof. Dr. Dr. Uwe Koch-Gromus __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing for R loop
On 10/11/2011 12:13 PM, Sandeep Patil wrote: I have an R script that consists of a for loop that repeats a process for many different files. I want to process this parallely on machine with multiple cores, is there any package for it ? Thanks ...I mostly use the foreach package... cheers, Paul -- Paul Hiemstra, Ph.D. Global Climate Division Royal Netherlands Meteorological Institute (KNMI) Wilhelminalaan 10 | 3732 GK | De Bilt | Kamer B 3.39 P.O. Box 201 | 3730 AE | De Bilt tel: +31 30 2206 494 http://intamap.geo.uu.nl/~paul http://nl.linkedin.com/pub/paul-hiemstra/20/30b/770 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Parallel processing for R loop
I have an R script that consists of a for loop that repeats a process for many different files. I want to process this parallely on machine with multiple cores, is there any package for it ? Thanks -- Sandeep R Patil [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] parallel processing for bridge and blasso
I'm running 10, 000 iterations each for the bridge and blasso. 3, 000 iterations roughly takes a week in a core-duo processor with 16GB RAM. I'll have access to a 6C processor machine and I came across the multicore package. Can I use multicore with the bridge and blasso function? That is, is parallel processing possible for the bridge and blasso functions by using the multicore package? Kind Regards Emily Deomano BSES Limited Q Australia Internet e-Mail Disclaimer PRIVILEGED - PRIVATE AND CONFIDENTIAL: This email and any files transmitted with it are intended solely for the use of the addressee(s) and may contain information, which is confidential or privileged. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the contents of this information is prohibited. In such case, you should destroy this message and kindly notify the sender by reply e-mail. The views and opinions expressed in this e-mail are those of the sender and do not necessarily reflect the views of the company. VIRUSES: Email transmission cannot be guaranteed to be secure or error free, as information may be intercepted, corrupted, lost, destroyed, arrive late or incomplete or contain viruses. This email and any files attached to it have been checked with virus detection software before transmission. You should nonetheless carry out your own virus check before opening any attachment. BSES Limited does not represent or warrant that files attached to this email are free from computer viruses or other defects and accepts no liability for any loss or damage that may be caused by software viruses __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing question ...
Hi David,
On Wed, Feb 9, 2011 at 10:11 AM, Robinson, David G dro...@sandia.gov wrote:
Steve,
Thanks for taking the time to look at the question. my apologies for the
confusing post. In an attempt to keep the post short, I seem to have
confused the issue.
The variable of interest in each iteration is the vector lambda and the
goal is to collect all the lambda vectors and characterize the statistics of
lambda over the course of the simulation (this is just a simply gibbs
sampler) . In the series processing world I simply use cbind to accumulate
the lambda vectors into an array called lambdas (as performed in commented
out commands).
What I am trying to do now is use a combination of foreach/dopar to do the
same type of accumulation.
I am not trying to capture any other variables from the loop except lambda.
As you suggested, I have tried removing the .combine argument and simply
collect the resulting list. Unfortunately, the lambda vectors don’t appear
in the resulting list.
Ok, so let's take a look at your code again (without the commented block):
lambdas- foreach(j=1:(2*burn_in), .combine=cbind) %dopar% {
N - update_N(min_sets, min_sets_indexes, lambda)
lambda - rgamma(K, shape=a+N, rate=bT)
lambda
if (j%%100==0) { print(j); print(lambda); print(N)}
}
Assuming you have all the vars defined properly. Note that your first
line in the loop passes a variable `lambda` into your update_N
function, then you reassign lambda to the result of your rgamma call.
I guess these two things are different, so why not name one of them
something else just in case (I've renamed it to LAMBDA below).
The last line of the block in your loop is an if-statement, which
essentially stops you from returning your lambda vector. Try:
lambdas- foreach(j=1:(2*burn_in), .combine=cbind) %dopar% {
N - update_N(min_sets, min_sets_indexes, LAMBDA)
lambda - rgamma(K, shape=a+N, rate=bT)
if (j%%100==0) { print(j); print(lambda); print(N)}
lambda
}
--
Steve Lianoglou
Graduate Student: Computational Systems Biology
| Memorial Sloan-Kettering Cancer Center
| Weill Medical College of Cornell University
Contact Info: http://cbio.mskcc.org/~lianos/contact
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing question ...
Steve,
Thanks for taking the time to look at the question. my apologies for the
confusing post. In an attempt to keep the post short, I seem to have confused
the issue.
The variable of interest in each iteration is the vector lambda and the goal
is to collect all the lambda vectors and characterize the statistics of lambda
over the course of the simulation (this is just a simply gibbs sampler) . In
the series processing world I simply use cbind to accumulate the lambda vectors
into an array called lambdas (as performed in commented out commands).
What I am trying to do now is use a combination of foreach/dopar to do the same
type of accumulation.
I am not trying to capture any other variables from the loop except lambda. As
you suggested, I have tried removing the .combine argument and simply collect
the resulting list. Unfortunately, the lambda vectors don't appear in the
resulting list.
Thanks again for taking the time to try to figure this out.
Cheers,
Dave
On 2/8/11 7:47 PM, Steve Lianoglou mailinglist.honey...@gmail.com wrote:
Hi,
On Tue, Feb 8, 2011 at 6:18 PM, Robinson, David G dro...@sandia.gov wrote:
I am experimenting with parallel processing using foreach and seem to be
missing something fundamental. Cool stuff. I've gone through the list and
seen a couple of closely related issues, but nothing I've tried seems to
work.
I know that the results from foreach are combined, but what if there is more
than one variable within the loop? Below is a snippet (non-functioning) of
code that I hope provides enough insight into what I am trying to do. The
commented out lines are what I would be doing (successfully) if I wasn't
trying to implement the %dopar% . The goal is to do statistics on the
sequence of lambda vectors that were originally accumulated in the matrix
lambdas using cbind.
Thanks in advance for any suggestions,
Dave
---snip
update_N - function(sets, indexes, lam) {
n - length(indexes)-1# count of events
N - rep(0, K) # count of failures per node
for (i in 1:n) {
nodes - as.numeric(sets[indexes[i]:(indexes[i+1]-1)])
node - resample(nodes, 1, prob=lam[nodes]/sum(lam[nodes]))
N[node] = N[node] + 1
}
N
}
lambdas- foreach(j=1:(2*burn_in), .combine=cbind) %dopar% {
N - update_N(min_sets, min_sets_indexes, lambda)
lambda - rgamma(K, shape=a+N, rate=bT)
lambda
if (j%%100==0) { print(j); print(lambda); print(N)}
#if (j burn_in) {
#lambdas - cbind(lambdas, lambda)
#}
}
---snip
Sorry -- I don't get what you're asking/trying to do.
Is it a coincidence that your commented block uses the same variable
name as the one you are assigning the result of foreach() to?
Essentially, foreach will work just like an lapply ... if you changed
foreach to lapply here, what do you expect that %dopar% {} block to
return after each iteration?
I'm not sure if this is what you're asking, but if you want to return
two elements per iteration in your loop, just return a list with two
elements, and post process it later.
I'd start by trying to remove your .combine=cbind param/argument from
the foreach() function and get your code running so you get the right
things returned as a normal list (or list of lists, if you want to
return 1 thing per foreach iteration). Once that's done, you can try
to auto 'cbind' your things if you think it's necessary.
Sorry if this isn't helpful .. it's not clear to me what you're trying
to do, so I'm kind of stabbing at the dark here.
--
Steve Lianoglou
Graduate Student: Computational Systems Biology
| Memorial Sloan-Kettering Cancer Center
| Weill Medical College of Cornell University
Contact Info: http://cbio.mskcc.org/~lianos/contact
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing question ...
Hi David,
I'm CC-ing R-help inorder to finish this one off ;-)
On Wed, Feb 9, 2011 at 10:59 AM, Robinson, David G dro...@sandia.gov wrote:
[snip]
One of you comments pointed me in the right direction and I found the
problem. I simply commented out the line if (j%%100==0) { ...print(N)}
and the original program ran fine. Not sure I understand why, but... it
runs.
[/snip]
It's because the last line of a block or function or whatever is
the implicit return value of that block/function (as you already know
-- the last line of your `update_N` function is `N`, which means
that's the value you want that function to return).
The last line of your the block inside your %dopar% { ... } was in
if-statement and not the value `lambda` that you wanted to return.
As a result the return value of your block was the result of that
if-statement. Keep in mind that in R, even `if` statements return
values, eg:
x - if (FALSE) {
1
} else {
2
}
In the case above, x will be set to 2.
Does that make it more clear now why your lambda vector wasn't being
returned (and further processed) after each iteration of your foreach
loop?
-steve
--
Steve Lianoglou
Graduate Student: Computational Systems Biology
| Memorial Sloan-Kettering Cancer Center
| Weill Medical College of Cornell University
Contact Info: http://cbio.mskcc.org/~lianos/contact
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Parallel processing question ...
I am experimenting with parallel processing using foreach and seem to be
missing something fundamental. Cool stuff. I've gone through the list and
seen a couple of closely related issues, but nothing I've tried seems to
work.
I know that the results from foreach are combined, but what if there is more
than one variable within the loop? Below is a snippet (non-functioning) of
code that I hope provides enough insight into what I am trying to do. The
commented out lines are what I would be doing (successfully) if I wasn't
trying to implement the %dopar% . The goal is to do statistics on the
sequence of lambda vectors that were originally accumulated in the matrix
lambdas using cbind.
Thanks in advance for any suggestions,
Dave
---snip
update_N - function(sets, indexes, lam) {
n - length(indexes)-1# count of events
N - rep(0, K) # count of failures per node
for (i in 1:n) {
nodes - as.numeric(sets[indexes[i]:(indexes[i+1]-1)])
node - resample(nodes, 1, prob=lam[nodes]/sum(lam[nodes]))
N[node] = N[node] + 1
}
N
}
lambdas- foreach(j=1:(2*burn_in), .combine=cbind) %dopar% {
N - update_N(min_sets, min_sets_indexes, lambda)
lambda - rgamma(K, shape=a+N, rate=bT)
lambda
if (j%%100==0) { print(j); print(lambda); print(N)}
#if (j burn_in) {
#lambdas - cbind(lambdas, lambda)
#}
}
---snip
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing question ...
Hi,
On Tue, Feb 8, 2011 at 6:18 PM, Robinson, David G dro...@sandia.gov wrote:
I am experimenting with parallel processing using foreach and seem to be
missing something fundamental. Cool stuff. I've gone through the list and
seen a couple of closely related issues, but nothing I've tried seems to
work.
I know that the results from foreach are combined, but what if there is more
than one variable within the loop? Below is a snippet (non-functioning) of
code that I hope provides enough insight into what I am trying to do. The
commented out lines are what I would be doing (successfully) if I wasn't
trying to implement the %dopar% . The goal is to do statistics on the
sequence of lambda vectors that were originally accumulated in the matrix
lambdas using cbind.
Thanks in advance for any suggestions,
Dave
---snip
update_N - function(sets, indexes, lam) {
n - length(indexes)-1 # count of events
N - rep(0, K) # count of failures per node
for (i in 1:n) {
nodes - as.numeric(sets[indexes[i]:(indexes[i+1]-1)])
node - resample(nodes, 1, prob=lam[nodes]/sum(lam[nodes]))
N[node] = N[node] + 1
}
N
}
lambdas- foreach(j=1:(2*burn_in), .combine=cbind) %dopar% {
N - update_N(min_sets, min_sets_indexes, lambda)
lambda - rgamma(K, shape=a+N, rate=bT)
lambda
if (j%%100==0) { print(j); print(lambda); print(N)}
# if (j burn_in) {
# lambdas - cbind(lambdas, lambda)
# }
}
---snip
Sorry -- I don't get what you're asking/trying to do.
Is it a coincidence that your commented block uses the same variable
name as the one you are assigning the result of foreach() to?
Essentially, foreach will work just like an lapply ... if you changed
foreach to lapply here, what do you expect that %dopar% {} block to
return after each iteration?
I'm not sure if this is what you're asking, but if you want to return
two elements per iteration in your loop, just return a list with two
elements, and post process it later.
I'd start by trying to remove your .combine=cbind param/argument from
the foreach() function and get your code running so you get the right
things returned as a normal list (or list of lists, if you want to
return 1 thing per foreach iteration). Once that's done, you can try
to auto 'cbind' your things if you think it's necessary.
Sorry if this isn't helpful .. it's not clear to me what you're trying
to do, so I'm kind of stabbing at the dark here.
--
Steve Lianoglou
Graduate Student: Computational Systems Biology
| Memorial Sloan-Kettering Cancer Center
| Weill Medical College of Cornell University
Contact Info: http://cbio.mskcc.org/~lianos/contact
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Parallel processing
1.what is the application to install for to speed up processing for multicore processor in windows environment? 2. how to compute time for executing a particular a code? [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Parallel processing
Hello Partha, Both questions are answered here: http://www.r-statistics.com/2010/04/parallel-multicore-processing-with-r-on-windows/ http://www.r-statistics.com/2010/04/parallel-multicore-processing-with-r-on-windows/I would also recommend you to have a look here: http://www.r-statistics.com/2010/09/using-the-plyr-1-2-package-parallel-processing-backend-with-windows/ There are claims for other packages to achieve this, but I wasn't able to make them work (I'd be glad to hear of better results by others) Best, Tal http://www.r-statistics.com/2010/04/parallel-multicore-processing-with-r-on-windows/ Contact Details:--- Contact me: tal.gal...@gmail.com | 972-52-7275845 Read me: www.talgalili.com (Hebrew) | www.biostatistics.co.il (Hebrew) | www.r-statistics.com (English) -- On Sun, Oct 10, 2010 at 2:58 PM, Partha Sinha pnsinh...@gmail.com wrote: 1.what is the application to install for to speed up processing for multicore processor in windows environment? 2. how to compute time for executing a particular a code? [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Parallel Processing and Linear Regression
Does anybody have any suggestions regarding applying standard regression packages lm(), hccm(), and others within a parallel environment? Most of the packages I've found only deal with iterative processes (bootstrap) or simple linear algebra. While the latter might help, I'd rather not program the estimation code. I'm currently using a IA-64 Teragrid system through UC San Diego. Alan [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
