Re: [R] meta analysis with repeated measure-designs?
Dear Gerrit,
Sorry. There was an error in my previous code. As a record, the
followings are the revised code.
Robust SE based on Hedges et al., (2010) Eq. 6 on Research
Synthesis Methods
rma.obj: object fitted by metafor()
cluster: indicator for clusters of studies
robustSE - function(rma.obj, cluster=NULL, CI=.95) {
# m: no. of clusters; assumed independent if not specified
# rma.obj$not.na: complete cases
if (is.null(cluster)) {
m=length(rma.obj$X[rma.obj$not.na,1])
} else {
m=nlevels(unique(as.factor(cluster[rma.obj$not.na])))
}
res2 - diag(residuals(rma.obj)^2)
X - rma.obj$X
b - rma.obj$b
W - diag(1/(rma.obj$vi+rma.obj$tau2)) # Use vi+tau2
meat - t(X) %*% W %*% res2 %*% W %*% X# W is symmetric
bread - solve( t(X) %*% W %*% X)
V.R - bread %*% meat %*% bread# Robust sampling covariance matrix
p - length(b) # no. of predictors
including intercept
se - sqrt( diag(V.R)*m/(m-p) )# small sample adjustment (Eq.7)
tval - b/se
pval - 2*(1-pt(abs(tval),df=(m-p)))
crit - qt( (1-CI)/2, df=(m-p), lower.tail=FALSE )
ci.lb - b-crit*se
ci.ub - b+crit*se
data.frame(estimate=b, se=se, tval=tval, pval=pval, ci.lb=ci.lb, ci.ub=ci.ub)
}
library(metafor)
data(dat.bcg)
### calculate log relative risks and corresponding sampling variances
dat - escalc(measure=RR, ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
dat - cbind(dat.bcg, dat)
### random-effects model
fit1 - rma(yi, vi, data=dat, method=DL)
summary(fit1)
estimate se zval pvalci.lbci.ub
-0.7141 0.1787 -3.9952 .0001 -1.0644 -0.3638 ***
robustSE(fit1)
estimatese tvalpval ci.lb ci.ub
intrcpt -0.7141172 0.1791445 -3.986265 0.001805797 -1.104439 -0.323795
### mixed-effects model with two moderators (absolute latitude and
publication year)
fit2 - rma(yi, vi, mods=cbind(ablat, year), data=dat, method=DL)
summary(fit2)
estimate se zvalpval ci.lbci.ub
intrcpt -1.2798 25.7550 -0.0497 0.9604 -51.7586 49.1990
ablat -0.0288 0.0090 -3.2035 0.0014 -0.0464 -0.0112 **
year 0.0008 0.0130 0.0594 0.9526 -0.0247 0.0262
robustSE(fit2)
estimate setvalpval
ci.lb ci.ub
intrcpt -1.2797914381 22.860353022 -0.05598301 0.956458098
-52.21583218 49.65624930
ablat -0.0287644840 0.007212163 -3.98832970 0.002566210
-0.04483418 -0.01269478
year 0.0007720798 0.011550188 0.06684565 0.948022174
-0.02496334 0.02650750
--
-
Mike W.L. Cheung Phone: (65) 6516-3702
Department of Psychology Fax: (65) 6773-1843
National University of Singapore
http://courses.nus.edu.sg/course/psycwlm/internet/
-
On Wed, Jun 16, 2010 at 4:47 PM, Mike Cheung mikewlche...@gmail.com wrote:
Dear Gerrit,
If the correlations of the dependent effect sizes are unknown, one
approach is to conduct the meta-analysis by assuming that the effect
sizes are independent. A robust standard error is then calculated to
adjust for the dependence. You may refer to Hedges et. al., (2010) for
more information. I have coded it here for reference.
Hedges, L. V., Tipton, E., Johnson, M. C. (2010). Robust variance
estimation in meta-regression with dependent effect size estimates.
Research Synthesis Methods, 1(1), 39-65. doi:10.1002/jrsm.5
Regards,
Mike
--
-
Mike W.L. Cheung Phone: (65) 6516-3702
Department of Psychology Fax: (65) 6773-1843
National University of Singapore
http://courses.nus.edu.sg/course/psycwlm/internet/
-
library(metafor)
Robust SE based on Hedges et al., (2010) Eq. 6
rma.obj: object fitted by metafor()
cluster: indicator for clusters of studies
robustSE - function(rma.obj, cluster=NULL, CI=.95) {
# m: no. of clusters; assumed independent if not specified
if (is.null(cluster)) {
m=nrow(rma.obj$X)
} else {
m=nlevels(unique(as.factor(cluster)))
}
res2 - diag(residuals(rma.obj)^2)
X - rma.obj$X
b - rma.obj$b
W - diag(1/(rma.obj$vi+rma.obj$tau2)) # Use vi+tau2
meat - t(X) %*% W %*% res2 %*% W %*% X # W is symmetric
bread - solve( t(X) %*% W %*% X)
V.R - bread %*% meat %*% bread # Robust sampling covariance
matrix
p - length(b) # no. of predictors
including intercept
se.R - sqrt( diag(V.R)*m/(m-p) ) # small sample adjustment (Eq.7)
t.R - b/se.R
p.R - 2*(1-pt(abs(t.R),df=(m-p)))
crit - qt( 1-CI/2, df=(m-p) )
ci.lb - b-crit*se.R
ci.ub - b+crit*se.R
data.frame(estimate=b, se.R=se.R, t.R=t.R, p.R=p.R, ci.lb=ci.lb, ci.ub=ci.ub)
}
## Example:
Re: [R] meta analysis with repeated measure-designs?
-boun...@r-project.org] On Behalf Of Gerrit Hirschfeld Sent: Saturday, June 12, 2010 12:45 To: r-help@r-project.org Subject: [R] meta analysis with repeated measure-designs? Dear all, I am trying to run a meta analysis of psycholinguistic reaction-time experiments with the meta package. The problem is that most of the studies have a within-subject designs and use repeated measures ANOVAs to analyze their data. So at present it seems that there are three non-optimal ways to run the analysis. 1. Using metacont() to estimate effect sizes and standard errors. But as the different sores are dependent this would result in biased estimators (Dunlap, 1996). Suppose I had the correlations of the measures (which I do not) would there by an option to use them in metacont() ? 2. Use metagen() with an effect size that is based on the reported F for the contrasts but has other disadvantages (Bakeman, 2005). The problem I am having with this is that I could not find a formular to compute the standard error of partial eta squared. Any Ideas? 3. Use metagen() with r computed from p-values (Rosenthal, 1994) as effect size with the problem that sample-size affects p as much as effect size. Is there a fourth way, or data showing that correlations can be neglected as long as they are assumed to be similar in the studies? Any ideas are much apprecciated. best regards Gerrit __ Gerrit Hirschfeld, Dipl.-Psych. Psychologisches Institut II Westfälische Wilhelms-Universität Fliednerstr. 21 48149 Münster Germany psycholinguistics.uni-muenster.de GerritHirschfeld.de Fon.: +49 (0) 251 83-31378 Fon.: +49 (0) 234 7960728 Fax.: +49 (0) 251 83-34104 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ Gerrit Hirschfeld, Dipl.-Psych. Psychologisches Institut II Westfälische Wilhelms-Universität Fliednerstr. 21 48149 Münster Germany psycholinguistics.uni-muenster.de GerritHirschfeld.de Fon.: +49 (0) 251 83-31378 Fon.: +49 (0) 234 7960728 Fax.: +49 (0) 251 83-34104 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] meta analysis with repeated measure-designs?
Hi, thanks for the references I will try the sensitivitiy-analysis in R and try out winbugs if that does not work (little afraid of switching programmes). I also had an idea for a reasonable estimate of the correlations. Some studies report both results from paired t-tests and means and SDs, and thus allow to calculate two estimates for d one based on M and SD alone the other on t. The difference between the two estimates should be systematically related to the correlations of measures. I will keep you posted, if I have a solution or hit a wall. efachristo and dank je wel! Gerrit On 12.06.2010, at 15:59, Viechtbauer Wolfgang (STAT) wrote: Dear Gerrit, the most appropriate approach for data of this type would be a proper multivariate meta-analytic model (along the lines of Kalaian Raudenbush, 1996). Since you do not know the correlations of the reaction time measurements across conditions for the within-subject designs, a simple solution is to guestimate those correlations and then conduct sensitivity analyses to make sure your conclusions do not depend on those guestimates. Best, -- Wolfgang Viechtbauerhttp://www.wvbauer.com/ Department of Methodology and StatisticsTel: +31 (0)43 388-2277 School for Public Health and Primary Care Office Location: Maastricht University, P.O. Box 616 Room B2.01 (second floor) 6200 MD Maastricht, The Netherlands Debyeplein 1 (Randwyck) Original Message From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of Gerrit Hirschfeld Sent: Saturday, June 12, 2010 12:45 To: r-help@r-project.org Subject: [R] meta analysis with repeated measure-designs? Dear all, I am trying to run a meta analysis of psycholinguistic reaction-time experiments with the meta package. The problem is that most of the studies have a within-subject designs and use repeated measures ANOVAs to analyze their data. So at present it seems that there are three non-optimal ways to run the analysis. 1. Using metacont() to estimate effect sizes and standard errors. But as the different sores are dependent this would result in biased estimators (Dunlap, 1996). Suppose I had the correlations of the measures (which I do not) would there by an option to use them in metacont() ? 2. Use metagen() with an effect size that is based on the reported F for the contrasts but has other disadvantages (Bakeman, 2005). The problem I am having with this is that I could not find a formular to compute the standard error of partial eta squared. Any Ideas? 3. Use metagen() with r computed from p-values (Rosenthal, 1994) as effect size with the problem that sample-size affects p as much as effect size. Is there a fourth way, or data showing that correlations can be neglected as long as they are assumed to be similar in the studies? Any ideas are much apprecciated. best regards Gerrit __ Gerrit Hirschfeld, Dipl.-Psych. Psychologisches Institut II Westfälische Wilhelms-Universität Fliednerstr. 21 48149 Münster Germany psycholinguistics.uni-muenster.de GerritHirschfeld.de Fon.: +49 (0) 251 83-31378 Fon.: +49 (0) 234 7960728 Fax.: +49 (0) 251 83-34104 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ Gerrit Hirschfeld, Dipl.-Psych. Psychologisches Institut II Westfälische Wilhelms-Universität Fliednerstr. 21 48149 Münster Germany psycholinguistics.uni-muenster.de GerritHirschfeld.de Fon.: +49 (0) 251 83-31378 Fon.: +49 (0) 234 7960728 Fax.: +49 (0) 251 83-34104 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] meta analysis with repeated measure-designs?
Dear all, I am trying to run a meta analysis of psycholinguistic reaction-time experiments with the meta package. The problem is that most of the studies have a within-subject designs and use repeated measures ANOVAs to analyze their data. So at present it seems that there are three non-optimal ways to run the analysis. 1. Using metacont() to estimate effect sizes and standard errors. But as the different sores are dependent this would result in biased estimators (Dunlap, 1996). Suppose I had the correlations of the measures (which I do not) would there by an option to use them in metacont() ? 2. Use metagen() with an effect size that is based on the reported F for the contrasts but has other disadvantages (Bakeman, 2005). The problem I am having with this is that I could not find a formular to compute the standard error of partial eta squared. Any Ideas? 3. Use metagen() with r computed from p-values (Rosenthal, 1994) as effect size with the problem that sample-size affects p as much as effect size. Is there a fourth way, or data showing that correlations can be neglected as long as they are assumed to be similar in the studies? Any ideas are much apprecciated. best regards Gerrit __ Gerrit Hirschfeld, Dipl.-Psych. Psychologisches Institut II Westfälische Wilhelms-Universität Fliednerstr. 21 48149 Münster Germany psycholinguistics.uni-muenster.de GerritHirschfeld.de Fon.: +49 (0) 251 83-31378 Fon.: +49 (0) 234 7960728 Fax.: +49 (0) 251 83-34104 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] meta analysis with repeated measure-designs?
Dear Gerrit, the most appropriate approach for data of this type would be a proper multivariate meta-analytic model (along the lines of Kalaian Raudenbush, 1996). Since you do not know the correlations of the reaction time measurements across conditions for the within-subject designs, a simple solution is to guestimate those correlations and then conduct sensitivity analyses to make sure your conclusions do not depend on those guestimates. Best, -- Wolfgang Viechtbauerhttp://www.wvbauer.com/ Department of Methodology and StatisticsTel: +31 (0)43 388-2277 School for Public Health and Primary Care Office Location: Maastricht University, P.O. Box 616 Room B2.01 (second floor) 6200 MD Maastricht, The Netherlands Debyeplein 1 (Randwyck) Original Message From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On Behalf Of Gerrit Hirschfeld Sent: Saturday, June 12, 2010 12:45 To: r-help@r-project.org Subject: [R] meta analysis with repeated measure-designs? Dear all, I am trying to run a meta analysis of psycholinguistic reaction-time experiments with the meta package. The problem is that most of the studies have a within-subject designs and use repeated measures ANOVAs to analyze their data. So at present it seems that there are three non-optimal ways to run the analysis. 1. Using metacont() to estimate effect sizes and standard errors. But as the different sores are dependent this would result in biased estimators (Dunlap, 1996). Suppose I had the correlations of the measures (which I do not) would there by an option to use them in metacont() ? 2. Use metagen() with an effect size that is based on the reported F for the contrasts but has other disadvantages (Bakeman, 2005). The problem I am having with this is that I could not find a formular to compute the standard error of partial eta squared. Any Ideas? 3. Use metagen() with r computed from p-values (Rosenthal, 1994) as effect size with the problem that sample-size affects p as much as effect size. Is there a fourth way, or data showing that correlations can be neglected as long as they are assumed to be similar in the studies? Any ideas are much apprecciated. best regards Gerrit __ Gerrit Hirschfeld, Dipl.-Psych. Psychologisches Institut II Westfälische Wilhelms-Universität Fliednerstr. 21 48149 Münster Germany psycholinguistics.uni-muenster.de GerritHirschfeld.de Fon.: +49 (0) 251 83-31378 Fon.: +49 (0) 234 7960728 Fax.: +49 (0) 251 83-34104 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
