Dear Gerrit, If the correlations of the dependent effect sizes are unknown, one approach is to conduct the meta-analysis by assuming that the effect sizes are independent. A robust standard error is then calculated to adjust for the dependence. You may refer to Hedges et. al., (2010) for more information. I have coded it here for reference.
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39-65. doi:10.1002/jrsm.5 Regards, Mike -- --------------------------------------------------------------------- Mike W.L. Cheung Phone: (65) 6516-3702 Department of Psychology Fax: (65) 6773-1843 National University of Singapore http://courses.nus.edu.sg/course/psycwlm/internet/ --------------------------------------------------------------------- library(metafor) #### Robust SE based on Hedges et al., (2010) Eq. 6 #### rma.obj: object fitted by metafor() #### cluster: indicator for clusters of studies robustSE <- function(rma.obj, cluster=NULL, CI=.95) { # m: no. of clusters; assumed independent if not specified if (is.null(cluster)) { m=nrow(rma.obj$X) } else { m=nlevels(unique(as.factor(cluster))) } res2 <- diag(residuals(rma.obj)^2) X <- rma.obj$X b <- rma.obj$b W <- diag(1/(rma.obj$vi+rma.obj$tau2)) # Use vi+tau2 meat <- t(X) %*% W %*% res2 %*% W %*% X # W is symmetric bread <- solve( t(X) %*% W %*% X) V.R <- bread %*% meat %*% bread # Robust sampling covariance matrix p <- length(b) # no. of predictors including intercept se.R <- sqrt( diag(V.R)*m/(m-p) ) # small sample adjustment (Eq.7) t.R <- b/se.R p.R <- 2*(1-pt(abs(t.R),df=(m-p))) crit <- qt( 1-CI/2, df=(m-p) ) ci.lb <- b-crit*se.R ci.ub <- b+crit*se.R data.frame(estimate=b, se.R=se.R, t.R=t.R, p.R=p.R, ci.lb=ci.lb, ci.ub=ci.ub) } ## Example: calculate log relative risks and corresponding sampling variances data(dat.bcg) dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg) dat <- cbind(dat.bcg, dat) ## random-effects model fit1 <- rma(yi, vi, data=dat, method="DL") summary(fit1) estimate se zval pval ci.lb ci.ub -0.7141 0.1787 -3.9952 <.0001 -1.0644 -0.3638 *** ## Robust SE robustSE(fit1) estimate se.R t.R p.R ci.lb ci.ub intrcpt -0.7141172 0.1791445 -3.986265 0.001805797 -0.725588 -0.7026465 ## mixed-effects model with two moderators (absolute latitude and publication year) fit2 <- rma(yi, vi, mods=cbind(ablat, year), data=dat, method="DL") summary(fit2) estimate se zval pval ci.lb ci.ub intrcpt -1.2798 25.7550 -0.0497 0.9604 -51.7586 49.1990 ablat -0.0288 0.0090 -3.2035 0.0014 -0.0464 -0.0112 ** year 0.0008 0.0130 0.0594 0.9526 -0.0247 0.0262 ### Robust SE robustSE(fit2) estimate se.R t.R p.R ci.lb intrcpt -1.2797914381 22.860353022 -0.05598301 0.956458098 -2.749670e+00 ablat -0.0287644840 0.007212163 -3.98832970 0.002566210 -2.922821e-02 year 0.0007720798 0.011550188 0.06684565 0.948022174 2.942412e-05 ci.ub intrcpt 0.190086881 ablat -0.028300755 year 0.001514735 On Mon, Jun 14, 2010 at 4:58 PM, Gerrit Hirschfeld <gerrit.hirschf...@uni-muenster.de> wrote: > Hi, > > thanks for the references I will try the sensitivitiy-analysis in R and try > out winbugs if that does not work (little afraid of switching programmes). > > I also had an idea for a reasonable estimate of the correlations. Some > studies report both results from paired t-tests and means and SDs, and thus > allow to calculate two estimates for d one based on M and SD alone the other > on t. The difference between the two estimates should be systematically > related to the correlations of measures. > > I will keep you posted, if I have a solution or hit a wall. > > efachristo and dank je wel! > > Gerrit > > > On 12.06.2010, at 15:59, Viechtbauer Wolfgang (STAT) wrote: > >> Dear Gerrit, >> >> the most appropriate approach for data of this type would be a proper >> multivariate meta-analytic model (along the lines of Kalaian & Raudenbush, >> 1996). Since you do not know the correlations of the reaction time >> measurements across conditions for the within-subject designs, a simple >> solution is to "guestimate" those correlations and then conduct sensitivity >> analyses to make sure your conclusions do not depend on those guestimates. >> >> Best, >> >> -- >> Wolfgang Viechtbauer http://www.wvbauer.com/ >> Department of Methodology and Statistics Tel: +31 (0)43 388-2277 >> School for Public Health and Primary Care Office Location: >> Maastricht University, P.O. Box 616 Room B2.01 (second floor) >> 6200 MD Maastricht, The Netherlands Debyeplein 1 (Randwyck) >> >> >> ----Original Message---- >> From: r-help-boun...@r-project.org [mailto:r-help-boun...@r-project.org] On >> Behalf Of Gerrit Hirschfeld Sent: Saturday, June 12, 2010 12:45 >> To: r-help@r-project.org >> Subject: [R] meta analysis with repeated measure-designs? >> >>> Dear all, >>> >>> I am trying to run a meta analysis of psycholinguistic reaction-time >>> experiments with the meta package. The problem is that most of the >>> studies have a within-subject designs and use repeated measures ANOVAs to >>> analyze their data. So at present it seems that there are three >>> non-optimal ways to run the analysis. >>> >>> 1. Using metacont() to estimate effect sizes and standard errors. But as >>> the different sores are dependent this would result in biased estimators >>> (Dunlap, 1996). Suppose I had the correlations of the measures (which I >>> do not) would there by an option to use them in metacont() ? >>> >>> 2. Use metagen() with an effect size that is based on the reported F for >>> the contrasts but has other disadvantages (Bakeman, 2005). The problem I >>> am having with this is that I could not find a formular to compute the >>> standard error of partial eta squared. Any Ideas? >>> >>> 3. Use metagen() with r computed from p-values (Rosenthal, 1994) as >>> effect size with the problem that sample-size affects p as much as effect >>> size. >>> >>> Is there a fourth way, or data showing that correlations can be neglected >>> as long as they are assumed to be similar in the studies? >>> Any ideas are much apprecciated. >>> >>> best regards >>> Gerrit >>> >>> ______________________________ >>> Gerrit Hirschfeld, Dipl.-Psych. >>> >>> Psychologisches Institut II >>> Westfälische Wilhelms-Universität >>> Fliednerstr. 21 >>> 48149 Münster >>> Germany >>> >>> psycholinguistics.uni-muenster.de >>> GerritHirschfeld.de >>> Fon.: +49 (0) 251 83-31378 >>> Fon.: +49 (0) 234 7960728 >>> Fax.: +49 (0) 251 83-34104 >>> >>> ______________________________________________ >>> R-help@r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide >>> http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. > > ______________________________ > Gerrit Hirschfeld, Dipl.-Psych. > > Psychologisches Institut II > Westfälische Wilhelms-Universität > Fliednerstr. 21 > 48149 Münster > Germany > > psycholinguistics.uni-muenster.de > GerritHirschfeld.de > Fon.: +49 (0) 251 83-31378 > Fon.: +49 (0) 234 7960728 > Fax.: +49 (0) 251 83-34104 > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.