Thanks to All,
The comments were very helpful; however, the the simulation is running very
slow. I reduced the number of loops (conditions) so I have 36 loops, and the
data-generation occurs 1000 times within each loop. At the end of each 1000
reps, I saved the summary (e.g., mean) of the reps to a single row vector,
and then saved it in a "fileshare" database. When the simulation is
finished, I rbind() the 36 rows of the database objects into a final
simulation result.
Thanks again,
Davood Tofighi
On Wed, Mar 5, 2008 at 8:44 AM, Paul Gilbert <[EMAIL PROTECTED]>
wrote:
> Davood Tofighi wrote:
> > Thanks for your reply. For each condition, I will have a matrix or data
> > frames of 1000 rows and 4 columns. I also have a total of 64 conditions
> for
> > now. So, in total, I will have 64 matrices or data frames of 1000 rows
> and 4
> > columns. The format of data I would like to store would be data frames
> or
> > matrices. I also would like to store the data for later use,
> I generally find it is better to store the seed and other data you need
> to reproduce the experiment, rather than trying to store the data (see,
> for example, package setRNG). If you save only the summary statistics
> you need, then you can usually do it in memory. (Be sure to assign
> variables for the statistics to their full size first and then populate
> them, rather than extending them at each step.) If you write things to
> files then it will slow down your simulation a lot. In fact, in most
> cases it will be quicker to re-run the experiment than it will be to
> read the data from disk.
>
> Paul
> > e.g., a plot of
> > the empirical distribution of the chi^2, or to compute the power of
> Chi^2
> > across 1000 reps for each condition.
> >
> > On Mon, Mar 3, 2008 at 7:03 PM, jim holtman <[EMAIL PROTECTED]> wrote:
> >
> >
> >> What is the format of the data you are storing (single value,
> >> multivalued vector, matrix, dataframe, ...)? This will help formulate
> >> a solution. What do you plan to do with the data? Are you going to
> >> do further analysis, write it to flat files, store it in a data base,
> >> etc.? How big are the data objects you are manipulating?
> >>
> >> On Mon, Mar 3, 2008 at 7:05 PM, Davood Tofighi <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> Dear All,
> >>>
> >>> I am running a Monte Carlo simulation study and have some questions on
> >>>
> >> how
> >>
> >>> to manage data storage efficiently at the end of each 1000 replication
> >>>
> >> loop.
> >>
> >>> I have three conditions coded using the FOR {} loops and a FOR loop
> that
> >>> generates data for each condition, performs analysis, and computes a
> >>> statistic 1000 times. Therefore, for each condition, I will have 1000
> >>> statistic values. My question is what's the best way to store the 1000
> >>> statistic for each condition. Any suggestion on how to manage such
> >>> simulation studies is greatly appreciated.
> >>> Thanks,
> >>>
> >>> --
> >>> Davood Tofighi
> >>> Department of Psychology
> >>> Arizona State University
> >>>
> >>> [[alternative HTML version deleted]]
> >>>
> >>> ______________________________________________
> >>> R-help@r-project.org mailing list
> >>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>> PLEASE do read the posting guide
> >>>
> >> http://www.R-project.org/posting-guide.html
> >>
> >>> and provide commented, minimal, self-contained, reproducible code.
> >>>
> >>>
> >>
> >> --
> >> Jim Holtman
> >> Cincinnati, OH
> >> +1 513 646 9390
> >>
> >> What is the problem you are trying to solve?
> >>
> >>
> >
> >
> >
> >
>
> ====================================================================================
>
> La version française suit le texte anglais.
>
>
> ------------------------------------------------------------------------------------
>
> This email may contain privileged and/or confidential information, and the
> Bank of
> Canada does not waive any related rights. Any distribution, use, or
> copying of this
> email or the information it contains by other than the intended recipient
> is
> unauthorized. If you received this email in error please delete it
> immediately from
> your system and notify the sender promptly by email that you have done so.
>
>
> ------------------------------------------------------------------------------------
>
> Le présent courriel peut contenir de l'information privilégiée ou
> confidentielle.
> La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute
> diffusion,
> utilisation ou copie de ce courriel ou des renseignements qu'il contient
> par une
> personne autre que le ou les destinataires désignés est interdite. Si vous
> recevez
> ce courriel par erreur, veuillez le supprimer immédiatement et envoyer
> sans délai à
> l'expéditeur un message électronique pour l'aviser que vous avez éliminé
> de votre
> ordinateur toute copie du courriel reçu.
>
--
Davood Tofighi
Department of Psychology
Arizona State University
P.O. BOX 871104
Tempe, AZ 85287-1104
Tel.:480-727-7884
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
