Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
Are the two models (f1 and f2) actually nested? Aside from that, it is strange that the output is exactly the same after you used REML=FALSE. The log likelihoods should have changed. Best, -- Wolfgang Viechtbauer Department of Methodology and Statistics School for Public Health and Primary Care University of Maastricht, The Netherlands http://www.wvbauer.com/ Original Message From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of Matt Killingsworth Sent: Friday, September 04, 2009 22:29 To: Bert Gunter Cc: r-help@r-project.org Subject: Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1 Hi Bert, Thank you for your note! I tried changing the REML default, and it still produces the same result (see below). Is that what you meant for me to try? Incidentally, I am using lmer() not lme() ### ORIGINAL ### f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 ### DO NOT USE REML ### f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i, REML = FALSE)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i, REML = FALSE)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 On Fri, Sep 4, 2009 at 4:18 PM, Bert Gunter gunter.ber...@gene.com wrote: My guess would be: Likelihood comparisons are not meaningful for objects fit using restricted maximum likelihood and with different fixed effects. (from ?anova.lme in the nlme package). Are you using the REML = TRUE default? Bert Gunter Genentech Nonclinical Statistics -Original Message- From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of rapton Sent: Friday, September 04, 2009 9:10 AM To: r-help@r-project.org Subject: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1 Hello, I am using R to analyze a large multilevel data set, using lmer() to model my data, and using anova() to compare the fit of various models. When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 -- __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
On Mon, 2009-09-07 at 10:23 +0200, Viechtbauer Wolfgang (STAT) wrote: Are the two models (f1 and f2) actually nested? Aside from that, it is strange that the output is exactly the same after you used REML=FALSE. The log likelihoods should have changed. I might be completely misremembering, but I recall a thread in R-SIG-Mixed where this was discussed and it was pointed out that anova(...) on mer objects extracts the ML information even if fitted using REML = TRUE. The log likelihoods of the models supplied to 'anova' are being extracted using REML = FALSE. So, if the above is correct, it does not surprise me that there is no difference. 'anova' was doing the right thing in both cases. See ?mer-class for more details, then try: logLik(f1, REML = FALSE) logLik(f1, REML = TRUE) logLik(f2, REML = FALSE) logLik(f2, REML = TRUE) 'anova' is calling logLik with REML = FALSE regardless of what you define in your model fitting call. HTH G Best, -- Wolfgang Viechtbauer Department of Methodology and Statistics School for Public Health and Primary Care University of Maastricht, The Netherlands http://www.wvbauer.com/ Original Message From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of Matt Killingsworth Sent: Friday, September 04, 2009 22:29 To: Bert Gunter Cc: r-help@r-project.org Subject: Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1 Hi Bert, Thank you for your note! I tried changing the REML default, and it still produces the same result (see below). Is that what you meant for me to try? Incidentally, I am using lmer() not lme() ### ORIGINAL ### f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 ### DO NOT USE REML ### f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i, REML = FALSE)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i, REML = FALSE)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 On Fri, Sep 4, 2009 at 4:18 PM, Bert Gunter gunter.ber...@gene.com wrote: My guess would be: Likelihood comparisons are not meaningful for objects fit using restricted maximum likelihood and with different fixed effects. (from ?anova.lme in the nlme package). Are you using the REML = TRUE default? Bert Gunter Genentech Nonclinical Statistics -Original Message- From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of rapton Sent: Friday, September 04, 2009 9:10 AM To: r-help@r-project.org Subject: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1 Hello, I am using R to analyze a large multilevel data set, using lmer() to model my data, and using anova() to compare the fit of various models. When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 -- __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~% Dr
Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
rapton wrote: Hello, I am using R to analyze a large multilevel data set, using lmer() to model my data, and using anova() to compare the fit of various models. When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 Your models are nest nestedit doesn't make sense to do. Alain - Dr. Alain F. Zuur First author of: 1. Analysing Ecological Data (2007). Zuur, AF, Ieno, EN and Smith, GM. Springer. 680 p. 2. Mixed effects models and extensions in ecology with R. (2009). Zuur, AF, Ieno, EN, Walker, N, Saveliev, AA, and Smith, GM. Springer. 3. A Beginner's Guide to R (2009). Zuur, AF, Ieno, EN, Meesters, EHWG. Springer Statistical consultancy, courses, data analysis and software Highland Statistics Ltd. 6 Laverock road UK - AB41 6FN Newburgh Email: highs...@highstat.com URL: www.highstat.com -- View this message in context: http://www.nabble.com/Using-anova%28f1%2C-f2%29-to-compare-lmer-models-yields-seemingly-erroneous-Chisq-%3D-0%2C-p-%3D-1-tp25297254p25333120.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
rapton wrote: Hello, I am using R to analyze a large multilevel data set, using lmer() to model my data, and using anova() to compare the fit of various models. When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 ** NOT ** nested sorrythe brain is going faster than the fingers. - Dr. Alain F. Zuur First author of: 1. Analysing Ecological Data (2007). Zuur, AF, Ieno, EN and Smith, GM. Springer. 680 p. 2. Mixed effects models and extensions in ecology with R. (2009). Zuur, AF, Ieno, EN, Walker, N, Saveliev, AA, and Smith, GM. Springer. 3. A Beginner's Guide to R (2009). Zuur, AF, Ieno, EN, Meesters, EHWG. Springer Statistical consultancy, courses, data analysis and software Highland Statistics Ltd. 6 Laverock road UK - AB41 6FN Newburgh Email: highs...@highstat.com URL: www.highstat.com -- View this message in context: http://www.nabble.com/Using-anova%28f1%2C-f2%29-to-compare-lmer-models-yields-seemingly-erroneous-Chisq-%3D-0%2C-p-%3D-1-tp25297254p25333148.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
On Mon, Sep 7, 2009 at 10:34 AM, Alain Zuurhighs...@highstat.com wrote: rapton wrote: Hello, I am using R to analyze a large multilevel data set, using lmer() to model my data, and using anova() to compare the fit of various models. When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. And, unless I'm missing something, also by the (misspecified) test. A large p-value indicates you have no evidence that the additional 19 parameters in f2 improve fit, which matches what the other methods suggested. However, as has been pointed out, the lack of nesting makes this a faulty LRT. This is made apparent by the fact that you get a test statistic outside the support of the chi-squared distribution (positive reals) (lambda - (-2)*(-22715 - (-23633))) [1] -1836 and since the test is uses right-tail probability, anova is not changing anything by moving the statistic to 0. pchisq(lambda, 19, lower = FALSE) [1] 1 pchisq(0, 19, lower = FALSE) [1] 1 To do the test properly the restricted (null) model must be a special case of the general (alternative) model (e.g., with the additional 19 parameters set to zero) which will result in the null model having a smaller likelihood, leading to a positive tests statistic. When that statistic is small you get a large p-value indicating a lack of evidence that the additional parameters improve fit... hth, Kingsford Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) Df AIC BIC logLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 ** NOT ** nested sorrythe brain is going faster than the fingers. - Dr. Alain F. Zuur First author of: 1. Analysing Ecological Data (2007). Zuur, AF, Ieno, EN and Smith, GM. Springer. 680 p. 2. Mixed effects models and extensions in ecology with R. (2009). Zuur, AF, Ieno, EN, Walker, N, Saveliev, AA, and Smith, GM. Springer. 3. A Beginner's Guide to R (2009). Zuur, AF, Ieno, EN, Meesters, EHWG. Springer Statistical consultancy, courses, data analysis and software Highland Statistics Ltd. 6 Laverock road UK - AB41 6FN Newburgh Email: highs...@highstat.com URL: www.highstat.com -- View this message in context: http://www.nabble.com/Using-anova%28f1%2C-f2%29-to-compare-lmer-models-yields-seemingly-erroneous-Chisq-%3D-0%2C-p-%3D-1-tp25297254p25333148.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
Thank you all for your insight! I am glad to hear, at least, that I am doing something incorrectly (since the results do not make sense), and I am very grateful for your attempts to remedy my very limited (and admittedly self-taught) understanding of multilevel models and R. As I mentioned in the problem statement, predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N), and the model estimates are very similar for the multilevel model as for OLS regression. Therefore, I am quite confident that predictor.1 comprises a much better model. I understand that several of you are saying that anova() cannot be used to compare these two multilevel models. Is there *any* way to compare two predictors to see which better predicts the outcome in a multilevel model? f1's lower AIC and BIC, and higher logLik are concordant with the idea that predictor.1 is superior to predictor.2, as best as I understand it, but is there any way to test whether that difference is statistically significant? The only function I can find online is anova() to compare models, but its output is nonsensical and, as you are all saying, it does not apply to my situation anyway. Interestingly, anova() seems to work if I arbitrarily subset my observations, but when I use all the observations anova() generates Chisq = 0. This is probably a red herring but I thought I would mention it in case it is not. Also, I concede that I am confused what you mean that the two models (f1 and f2) are not nested, and therefore anova() cannot be used. What would be an example of a nested model: comparing predictor.1 to a model with both predictor.1 and predictor.2? Surely there must also be a way to compare the predictive power of predictor.1 and predictor.2 to each other in a zero-order sense, but I am at a loss to identify it. Alain Zuur wrote: rapton wrote: When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 Your models are nest nestedit doesn't make sense to do. Alain -- View this message in context: http://www.nabble.com/Using-anova%28f1%2C-f2%29-to-compare-lmer-models-yields-seemingly-erroneous-Chisq-%3D-0%2C-p-%3D-1-tp25297254p25338046.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
Hello, I am using R to analyze a large multilevel data set, using lmer() to model my data, and using anova() to compare the fit of various models. When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 -- View this message in context: http://www.nabble.com/Using-anova%28f1%2C-f2%29-to-compare-lmer-models-yields-seemingly-erroneous-Chisq-%3D-0%2C-p-%3D-1-tp25297254p25297254.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
My guess would be: Likelihood comparisons are not meaningful for objects fit using restricted maximum likelihood and with different fixed effects. (from ?anova.lme in the nlme package). Are you using the REML = TRUE default? Bert Gunter Genentech Nonclinical Statistics -Original Message- From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of rapton Sent: Friday, September 04, 2009 9:10 AM To: r-help@r-project.org Subject: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1 Hello, I am using R to analyze a large multilevel data set, using lmer() to model my data, and using anova() to compare the fit of various models. When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 -- View this message in context: http://www.nabble.com/Using-anova%28f1%2C-f2%29-to-compare-lmer-models-yield s-seemingly-erroneous-Chisq-%3D-0%2C-p-%3D-1-tp25297254p25297254.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1
Hi Bert, Thank you for your note! I tried changing the REML default, and it still produces the same result (see below). Is that what you meant for me to try? Incidentally, I am using lmer() not lme() ### ORIGINAL ### f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 ### DO NOT USE REML ### f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i, REML = FALSE)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i, REML = FALSE)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 On Fri, Sep 4, 2009 at 4:18 PM, Bert Gunter gunter.ber...@gene.com wrote: My guess would be: Likelihood comparisons are not meaningful for objects fit using restricted maximum likelihood and with different fixed effects. (from ?anova.lme in the nlme package). Are you using the REML = TRUE default? Bert Gunter Genentech Nonclinical Statistics -Original Message- From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of rapton Sent: Friday, September 04, 2009 9:10 AM To: r-help@r-project.org Subject: [R] Using anova(f1, f2) to compare lmer models yields seemingly erroneous Chisq = 0, p = 1 Hello, I am using R to analyze a large multilevel data set, using lmer() to model my data, and using anova() to compare the fit of various models. When I run two models, the output of each model is generated correctly as far as I can tell (e.g. summary(f1) and summary(f2) for the multilevel model output look perfectly reasonable), and in this case (see below) predictor.1 explains vastly more variance in outcome than predictor.2 (R2 = 15% vs. 5% in OLS regression, with very large N). What I am utterly puzzled by is that when I run an anova comparing the two multilevel model fits, the Chisq comes back as 0, with p = 1. I am pretty sure that fit #1 (f1) is a much better predictor of the outcome than f2, which is reflected in the AIC, BIC , and logLik values. Why might anova be giving me this curious output? How can I fix it? I am sure I am making a dumb error somewhere, but I cannot figure out what it is. Any help or suggestions would be greatly appreciated! -Matt f1 - (lmer(outcome ~ predictor.1 + (1 | person), data=i)) f2 - (lmer(outcome ~ predictor.2 + (1 | person), data=i)) anova(f1, f2) Data: i Models: f1: outcome ~ predictor.1 + (1 | person) f2: outcome ~ predictor.2 + (1 | person) DfAIC BIClogLik Chisq Chi Df Pr(Chisq) f1 6 45443 45489 -22715 f2 25 47317 47511 -23633 0 19 1 -- View this message in context: http://www.nabble.com/Using-anova%28f1%2C-f2%29-to-compare-lmer-models-yield s-seemingly-erroneous-Chisq-%3D-0%2C-p-%3D-1-tp25297254p25297254.htmlhttp://www.nabble.com/Using-anova%28f1%2C-f2%29-to-compare-lmer-models-yield%0As-seemingly-erroneous-Chisq-%3D-0%2C-p-%3D-1-tp25297254p25297254.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
