Re: [R] Fwd: Variance Components in R

[Spencer Graves]

  Burt Gunter just reminded me that the completion time could also 
be affected by the numbers of levels of each of the factors, especially 
random effects:  With N records, any variance components / mixed model 
software using MLE or REML will have to invert repeatedly an N x N 
matrix for the covariance structure of the random effects and noise.  If 
the software recognizes your design as having some simple structure, 
this can be quite fast;  otherwise, it could be a Herculean task.  In 
your case with N = 9500 records, just one copy of this covariance matrix 
could consume a substantial portion of 1GB RAM.  I compute 
8*9500*(9500-1)/2 = 361Mbytes. 

  However, any software that recognizes special structure in your 
design may be able to do the required computations without ever 
constructing a matrix this large.  I would say that it's still worth a 
try in R on your laptop or on the machine with 1GB RAM:  'lmer' might 
recognize special structure that neither of the other two do (and vice 
versa). 

  Hope this helps. 
  Spencer Graves

Iuri Gavronski wrote:
> We have tried on many machines, from my laptop to a dual core Intel 
> processor with 1GB of RAM.
>
> On 8/17/06, *Spencer Graves* < [EMAIL PROTECTED] 
> > wrote:
>
> Hi, Iuri:
>
>   How much RAM and how fast a microprocessor (and what version of
> Windows) do you have?  You might still try it in R under Windows.  The
> results might be comparable or dramatically better in R than in
> SPSS or
> SAS.
>
>   hope this helps.
>   Spencer Graves
>
> Iuri Gavronski wrote:
> > 9500 records. It didn`t run in SPSS or SAS on Windows machines,
> so I am
> > trying to convert the SPSS script to R to run in a RISC station
> at the
> > university.
> >
> > On 8/17/06, Doran, Harold < [EMAIL PROTECTED]
> > wrote:
> >
> >> Iuri:
> >>
> >> The lmer function is optimal for large data with crossed random
> effects.
> >> How large are your data?
> >>
> >>
> >>> -Original Message-
> >>> From: [EMAIL PROTECTED]
> 
> >>> [mailto: [EMAIL PROTECTED]
> ] On Behalf Of Iuri Gavronski
> >>> Sent: Thursday, August 17, 2006 11:08 AM
> >>> To: Spencer Graves
> >>> Cc: r-help@stat.math.ethz.ch 
> >>> Subject: Re: [R] Variance Components in R
> >>>
> >>> Thank you for your reply.
> >>> VARCOMP is available at SPSS advanced models, I'm not sure
> >>> for how long it exists... I only work with SPSS for the last
> >>> 4 years...
> >>> My model only has crossed random effects, what perhaps would
> >>> drive me to lmer().
> >>> However, as I have unbalanced data (why it is normally called
> >>> 'unbalanced design'? the data was not intended to be
> >>> unbalanced, only I could not get responses for all cells...),
> >>> I'm afraid that REML would take too much CPU, memory and time
> >>> to execute, and MINQUE would be faster and provide similar
> >>> variance estimates (please, correct me if I'm wrong on that
> point).
> >>> I only found MINQUE on the maanova package, but as my study
> >>> is very far from genetics, I'm not sure I can use this package.
> >>> Any comment would be appreciated.
> >>> Iuri
> >>>
> >>> On 8/16/06, Spencer Graves < [EMAIL PROTECTED]
> > wrote:
> >>>
>    I used SPSS over 25 years ago, but I don't recall
> 
> >>> ever fitting a
> >>>
>  variance components model with it.  Are all your random
> 
> >>> effects nested?
> >>>
>  If they were, I would recommend you use 'lme' in the 'nlme'
> package.
>  However, if you have crossed random effects, I suggest you
> 
> >>> try 'lmer'
> >>>
>  associated with the 'lme4' package.
> 
>    For 'lmer', documentation is available in Douglas
> 
> >>> Bates. Fitting
> >>>
>  linear mixed models in R. /R News/, 5(1):27-30, May 2005
>  (www.r-project.org  ->
> newsletter).  I also recommend you try the
>  vignette available with the 'mlmRev' package (see, e.g.,
>  http://finzi.psych.upenn.edu/R/Rhelp02a/archive/81375.html
> ).
> 
> Excellent documentation for both 'lme' (and indirectly for
>  'lmer') is available in Pinheiro and Bates (2000)
> 
> >>> Mixed-Effects Models
> >>>
>  in S and S-Plus (Springer).  I have personally recommended
> 
> >>> this book
> >>>
>  so many times on this listserve that I just now got 234 hits for
>  RSiteSearch("graves pinheiro").  Pleas

Re: [R] Fwd: Variance Components in R

[Iuri Gavronski]

We have tried on many machines, from my laptop to a dual core Intel
processor with 1GB of RAM.

On 8/17/06, Spencer Graves <[EMAIL PROTECTED]> wrote:
>
> Hi, Iuri:
>
>   How much RAM and how fast a microprocessor (and what version of
> Windows) do you have?  You might still try it in R under Windows.  The
> results might be comparable or dramatically better in R than in SPSS or
> SAS.
>
>   hope this helps.
>   Spencer Graves
>
> Iuri Gavronski wrote:
> > 9500 records. It didn`t run in SPSS or SAS on Windows machines, so I am
> > trying to convert the SPSS script to R to run in a RISC station at the
> > university.
> >
> > On 8/17/06, Doran, Harold <[EMAIL PROTECTED]> wrote:
> >
> >> Iuri:
> >>
> >> The lmer function is optimal for large data with crossed random
> effects.
> >> How large are your data?
> >>
> >>
> >>> -Original Message-
> >>> From: [EMAIL PROTECTED]
> >>> [mailto:[EMAIL PROTECTED] On Behalf Of Iuri Gavronski
> >>> Sent: Thursday, August 17, 2006 11:08 AM
> >>> To: Spencer Graves
> >>> Cc: r-help@stat.math.ethz.ch
> >>> Subject: Re: [R] Variance Components in R
> >>>
> >>> Thank you for your reply.
> >>> VARCOMP is available at SPSS advanced models, I'm not sure
> >>> for how long it exists... I only work with SPSS for the last
> >>> 4 years...
> >>> My model only has crossed random effects, what perhaps would
> >>> drive me to lmer().
> >>> However, as I have unbalanced data (why it is normally called
> >>> 'unbalanced design'? the data was not intended to be
> >>> unbalanced, only I could not get responses for all cells...),
> >>> I'm afraid that REML would take too much CPU, memory and time
> >>> to execute, and MINQUE would be faster and provide similar
> >>> variance estimates (please, correct me if I'm wrong on that point).
> >>> I only found MINQUE on the maanova package, but as my study
> >>> is very far from genetics, I'm not sure I can use this package.
> >>> Any comment would be appreciated.
> >>> Iuri
> >>>
> >>> On 8/16/06, Spencer Graves <[EMAIL PROTECTED]> wrote:
> >>>
>    I used SPSS over 25 years ago, but I don't recall
> 
> >>> ever fitting a
> >>>
>  variance components model with it.  Are all your random
> 
> >>> effects nested?
> >>>
>  If they were, I would recommend you use 'lme' in the 'nlme' package.
>  However, if you have crossed random effects, I suggest you
> 
> >>> try 'lmer'
> >>>
>  associated with the 'lme4' package.
> 
>    For 'lmer', documentation is available in Douglas
> 
> >>> Bates. Fitting
> >>>
>  linear mixed models in R. /R News/, 5(1):27-30, May 2005
>  (www.r-project.org -> newsletter).  I also recommend you try the
>  vignette available with the 'mlmRev' package (see, e.g.,
>  http://finzi.psych.upenn.edu/R/Rhelp02a/archive/81375.html).
> 
> Excellent documentation for both 'lme' (and indirectly for
>  'lmer') is available in Pinheiro and Bates (2000)
> 
> >>> Mixed-Effects Models
> >>>
>  in S and S-Plus (Springer).  I have personally recommended
> 
> >>> this book
> >>>
>  so many times on this listserve that I just now got 234 hits for
>  RSiteSearch("graves pinheiro").  Please don't hesitate to pass this
>  recommendation to your university library.  This book is
> 
> >>> the primary
> >>>
>  documentation for the 'nlme' package, which is part of the
> 
> >>> standard R
> >>>
>  distribution.  A subdirectory "~library\nlme\scripts" of your R
>  installation includes files named "ch01.R", "ch02.R", ...,
> 
> >>> "ch06.R",
> >>>
>  "ch08.R", containing the R scripts described in the book.  These R
>  script files make it much easier and more enjoyable to study that
>  book, because they make it much easier to try the commands
> 
> >>> described
> >>>
>  in the book, one line at a time, testing modifications to check you
>  comprehension, etc.  In addition to avoiding problems with
>  typographical errors, it also automatically overcomes a few
> 
> >>> minor but
> >>>
>  substantive changes in the notation between S-Plus and R.
> 
>    Also, the "MINQUE" method has been obsolete for over
> 
> >>> 25 years.
> >>>
>  I recommend you use method = "REML" except for when you want to
>  compare two nested models with different fixed effects;  in
> 
> >>> that case,
> >>>
>  you should use method = "ML", as explained in Pinheiro and
> 
> >>> Bates (2000).
> >>>
>    Hope this helps.
>    Spencer Graves
> 
>  Iuri Gavronski wrote:
> 
> > Hi,
> >
> > I'm trying to fit a model using variance components in R, but if
> > very new on it, so I'm asking for your help.
> >
> > I have imported the SPSS database onto R, but I don't know how to
> > convert the commands... the SPSS commands I'm trying to
> >
> >>> convert are:
> >>>
> > VARCOMP
> >RATING BY CHAIN SECT

Re: [R] Fwd: Variance Components in R

[Spencer Graves]

Hi, Iuri: 

  How much RAM and how fast a microprocessor (and what version of 
Windows) do you have?  You might still try it in R under Windows.  The 
results might be comparable or dramatically better in R than in SPSS or 
SAS. 

  hope this helps.
  Spencer Graves

Iuri Gavronski wrote:
> 9500 records. It didn`t run in SPSS or SAS on Windows machines, so I am
> trying to convert the SPSS script to R to run in a RISC station at the
> university.
>
> On 8/17/06, Doran, Harold <[EMAIL PROTECTED]> wrote:
>   
>> Iuri:
>>
>> The lmer function is optimal for large data with crossed random effects.
>> How large are your data?
>>
>> 
>>> -Original Message-
>>> From: [EMAIL PROTECTED]
>>> [mailto:[EMAIL PROTECTED] On Behalf Of Iuri Gavronski
>>> Sent: Thursday, August 17, 2006 11:08 AM
>>> To: Spencer Graves
>>> Cc: r-help@stat.math.ethz.ch
>>> Subject: Re: [R] Variance Components in R
>>>
>>> Thank you for your reply.
>>> VARCOMP is available at SPSS advanced models, I'm not sure
>>> for how long it exists... I only work with SPSS for the last
>>> 4 years...
>>> My model only has crossed random effects, what perhaps would
>>> drive me to lmer().
>>> However, as I have unbalanced data (why it is normally called
>>> 'unbalanced design'? the data was not intended to be
>>> unbalanced, only I could not get responses for all cells...),
>>> I'm afraid that REML would take too much CPU, memory and time
>>> to execute, and MINQUE would be faster and provide similar
>>> variance estimates (please, correct me if I'm wrong on that point).
>>> I only found MINQUE on the maanova package, but as my study
>>> is very far from genetics, I'm not sure I can use this package.
>>> Any comment would be appreciated.
>>> Iuri
>>>
>>> On 8/16/06, Spencer Graves <[EMAIL PROTECTED]> wrote:
>>>   
   I used SPSS over 25 years ago, but I don't recall
 
>>> ever fitting a
>>>   
 variance components model with it.  Are all your random
 
>>> effects nested?
>>>   
 If they were, I would recommend you use 'lme' in the 'nlme' package.
 However, if you have crossed random effects, I suggest you
 
>>> try 'lmer'
>>>   
 associated with the 'lme4' package.

   For 'lmer', documentation is available in Douglas
 
>>> Bates. Fitting
>>>   
 linear mixed models in R. /R News/, 5(1):27-30, May 2005
 (www.r-project.org -> newsletter).  I also recommend you try the
 vignette available with the 'mlmRev' package (see, e.g.,
 http://finzi.psych.upenn.edu/R/Rhelp02a/archive/81375.html).

Excellent documentation for both 'lme' (and indirectly for
 'lmer') is available in Pinheiro and Bates (2000)
 
>>> Mixed-Effects Models
>>>   
 in S and S-Plus (Springer).  I have personally recommended
 
>>> this book
>>>   
 so many times on this listserve that I just now got 234 hits for
 RSiteSearch("graves pinheiro").  Please don't hesitate to pass this
 recommendation to your university library.  This book is
 
>>> the primary
>>>   
 documentation for the 'nlme' package, which is part of the
 
>>> standard R
>>>   
 distribution.  A subdirectory "~library\nlme\scripts" of your R
 installation includes files named "ch01.R", "ch02.R", ...,
 
>>> "ch06.R",
>>>   
 "ch08.R", containing the R scripts described in the book.  These R
 script files make it much easier and more enjoyable to study that
 book, because they make it much easier to try the commands
 
>>> described
>>>   
 in the book, one line at a time, testing modifications to check you
 comprehension, etc.  In addition to avoiding problems with
 typographical errors, it also automatically overcomes a few
 
>>> minor but
>>>   
 substantive changes in the notation between S-Plus and R.

   Also, the "MINQUE" method has been obsolete for over
 
>>> 25 years.
>>>   
 I recommend you use method = "REML" except for when you want to
 compare two nested models with different fixed effects;  in
 
>>> that case,
>>>   
 you should use method = "ML", as explained in Pinheiro and
 
>>> Bates (2000).
>>>   
   Hope this helps.
   Spencer Graves

 Iuri Gavronski wrote:
 
> Hi,
>
> I'm trying to fit a model using variance components in R, but if
> very new on it, so I'm asking for your help.
>
> I have imported the SPSS database onto R, but I don't know how to
> convert the commands... the SPSS commands I'm trying to
>   
>>> convert are:
>>>   
> VARCOMP
>RATING BY CHAIN SECTOR RESP ASPECT ITEM
>/RANDOM = CHAIN SECTOR RESP ASPECT ITEM
>/METHOD = MINQUE (1)
>/DESIGN = CHAIN SECTOR RESP ASPECT ITEM
>   

[R] Fwd: Variance Components in R

[Iuri Gavronski]

9500 records. It didn`t run in SPSS or SAS on Windows machines, so I am
trying to convert the SPSS script to R to run in a RISC station at the
university.

On 8/17/06, Doran, Harold <[EMAIL PROTECTED]> wrote:
>
> Iuri:
>
> The lmer function is optimal for large data with crossed random effects.
> How large are your data?
>
> > -Original Message-
> > From: [EMAIL PROTECTED]
> > [mailto:[EMAIL PROTECTED] On Behalf Of Iuri Gavronski
> > Sent: Thursday, August 17, 2006 11:08 AM
> > To: Spencer Graves
> > Cc: r-help@stat.math.ethz.ch
> > Subject: Re: [R] Variance Components in R
> >
> > Thank you for your reply.
> > VARCOMP is available at SPSS advanced models, I'm not sure
> > for how long it exists... I only work with SPSS for the last
> > 4 years...
> > My model only has crossed random effects, what perhaps would
> > drive me to lmer().
> > However, as I have unbalanced data (why it is normally called
> > 'unbalanced design'? the data was not intended to be
> > unbalanced, only I could not get responses for all cells...),
> > I'm afraid that REML would take too much CPU, memory and time
> > to execute, and MINQUE would be faster and provide similar
> > variance estimates (please, correct me if I'm wrong on that point).
> > I only found MINQUE on the maanova package, but as my study
> > is very far from genetics, I'm not sure I can use this package.
> > Any comment would be appreciated.
> > Iuri
> >
> > On 8/16/06, Spencer Graves <[EMAIL PROTECTED]> wrote:
> > >
> > >   I used SPSS over 25 years ago, but I don't recall
> > ever fitting a
> > > variance components model with it.  Are all your random
> > effects nested?
> > > If they were, I would recommend you use 'lme' in the 'nlme' package.
> > > However, if you have crossed random effects, I suggest you
> > try 'lmer'
> > > associated with the 'lme4' package.
> > >
> > >   For 'lmer', documentation is available in Douglas
> > Bates. Fitting
> > > linear mixed models in R. /R News/, 5(1):27-30, May 2005
> > > (www.r-project.org -> newsletter).  I also recommend you try the
> > > vignette available with the 'mlmRev' package (see, e.g.,
> > > http://finzi.psych.upenn.edu/R/Rhelp02a/archive/81375.html).
> > >
> > >Excellent documentation for both 'lme' (and indirectly for
> > > 'lmer') is available in Pinheiro and Bates (2000)
> > Mixed-Effects Models
> > > in S and S-Plus (Springer).  I have personally recommended
> > this book
> > > so many times on this listserve that I just now got 234 hits for
> > > RSiteSearch("graves pinheiro").  Please don't hesitate to pass this
> > > recommendation to your university library.  This book is
> > the primary
> > > documentation for the 'nlme' package, which is part of the
> > standard R
> > > distribution.  A subdirectory "~library\nlme\scripts" of your R
> > > installation includes files named "ch01.R", "ch02.R", ...,
> > "ch06.R",
> > > "ch08.R", containing the R scripts described in the book.  These R
> > > script files make it much easier and more enjoyable to study that
> > > book, because they make it much easier to try the commands
> > described
> > > in the book, one line at a time, testing modifications to check you
> > > comprehension, etc.  In addition to avoiding problems with
> > > typographical errors, it also automatically overcomes a few
> > minor but
> > > substantive changes in the notation between S-Plus and R.
> > >
> > >   Also, the "MINQUE" method has been obsolete for over
> > 25 years.
> > > I recommend you use method = "REML" except for when you want to
> > > compare two nested models with different fixed effects;  in
> > that case,
> > > you should use method = "ML", as explained in Pinheiro and
> > Bates (2000).
> > >
> > >   Hope this helps.
> > >   Spencer Graves
> > >
> > > Iuri Gavronski wrote:
> > > > Hi,
> > > >
> > > > I'm trying to fit a model using variance components in R, but if
> > > > very new on it, so I'm asking for your help.
> > > >
> > > > I have imported the SPSS database onto R, but I don't know how to
> > > > convert the commands... the SPSS commands I'm trying to
> > convert are:
> > > > VARCOMP
> > > >RATING BY CHAIN SECTOR RESP ASPECT ITEM
> > > >/RANDOM = CHAIN SECTOR RESP ASPECT ITEM
> > > >/METHOD = MINQUE (1)
> > > >/DESIGN = CHAIN SECTOR RESP ASPECT ITEM
> > > >SECTOR*RESP SECTOR*ASPECT SECTOR*ITEM CHAIN*RESP
> > > > CHAIN*ASPECT CHAIN*ITEM RESP*ASPECT RESP*ITEM
> > > >SECTOR*RESP*ASPECT SECTOR*RESP*ITEM
> > CHAIN*RESP*ASPECT
> > > >/INTERCEPT = INCLUDE.
> > > >
> > > > VARCOMP
> > > >RATING BY CHAIN SECTOR RESP ASPECT ITEM
> > > >/RANDOM = CHAIN SECTOR RESP ASPECT ITEM
> > > >/METHOD = REML
> > > >/DESIGN = CHAIN SECTOR RESP ASPECT ITEM
> > > >SECTOR*RESP SECTOR*ASPECT SECTOR*ITEM CHAIN*RESP
> > > > CHAIN*ASPECT CHAIN*ITEM RESP*ASPECT RESP*ITEM
> > > >SECTOR*RESP*ASPECT SECTOR*RESP*ITEM
> > CHAIN*RESP*ASPECT
> > > >/INTERCEPT =