[R] Post-hoc tests for repeated measures in balanced experimental design
Hi, I am performing experiments in the field of visual perception where we often apply balanced designs. Within a group of normal subjects, we vary different stimulus conditions (like contrast, luminance, temporal frequency of stimulation) and derive some psychophysical or electrophysiological results from our subjects. Often, the main question is to test the effect of these parameters on the recorded results. When switching from Statview (old software on Mac computers, no longer supported) to R I learned that the problem is translated to a repeated measures ANOVA via ano - aov(d$Result~ d$Condition1*d$Condition2 + Error(d$Subject/(d $Condition1*d$Condition2), data=d)) However, there are problems in performing post-hoc tests due to the intraindividual correlation of the repeated measures data. So I started intense online searches for a good solution in R, found snippets of R-code here and there. One problem was that many contributions offered help for the first step to replace the aov procedure with a call to lme in the nlme package, but did not perform post-hoc tests. Other contributions showed examples with only one within-subjects factors. Finally, Achim Zeileis (thanks again!) helped me with the following approach. 1. Use the nlme-Package and calculate a model with both within- subjects effects Condition1 and Condition2 library(nlme) d.lme - lme(Result~ Condition1*Condition2,data=d,random= ~1 | Subject) 2. Finally, the multcomp-Package can be used to perform post-hoc tests for Condition1 and Condition2 library(multcomp) print(summary(glht(d.lme, linfct=mcp(Condition1=Tukey print(summary(glht(d.lme, linfct=mcp(Condition2=Tukey My problems and questions are 1) When applying this solution to my collection of data I found several cases where the standard repeated-measures ANOVA showed a highly significant effect for both factors, e. g. ano - aov(d$wPatternPulseSNR~ d$Bedingung*d$Felder + Error(d$VPerson/ (d$Bedingung*d$Felder), data=d)) Error: d$VPerson Df Sum Sq Mean Sq F value Pr(F) Residuals 11 458.22 41.66 Error: d$VPerson:d$Bedingung Df Sum Sq Mean Sq F valuePr(F) d$Bedingung 4 364.58 91.14 7.4429 0.0001140 *** Residuals 44 538.81 12.25 Error: d$VPerson:d$Felder Df Sum Sq Mean Sq F valuePr(F) d$Felder 5 464.17 92.83 8.3957 5.953e-06 *** Residuals 55 608.16 11.06 but the multcomp-results indicated no significant post-hoc differences between any pair of values for both factors (here the values for the factor Felder as example) Fit: lme.formula(fixed = wPatternPulseSNR ~ Bedingung * Felder, data = d, random = ~1 | VPerson) Linear Hypotheses: Estimate Std. Error z value Pr(|z|) Feld2 - Feld1 == 0 1.897480.84245 2.2520.214 Feld3 - Feld1 == 0 1.413830.84245 1.6780.546 Feld4 - Feld1 == 0 1.489450.84245 1.7680.487 Feld5 - Feld1 == 0 -0.111330.84245 -0.1321.000 Feld6 - Feld1 == 0 0.024720.84245 0.0291.000 Feld3 - Feld2 == 0 -0.483660.84245 -0.5740.993 Feld4 - Feld2 == 0 -0.408030.84245 -0.4840.997 Feld5 - Feld2 == 0 -2.008820.84245 -2.3850.162 Feld6 - Feld2 == 0 -1.872770.84245 -2.2230.227 Feld4 - Feld3 == 0 0.075620.84245 0.0901.000 Feld5 - Feld3 == 0 -1.525160.84245 -1.8100.459 Feld6 - Feld3 == 0 -1.389110.84245 -1.6490.566 Feld5 - Feld4 == 0 -1.600780.84245 -1.9000.402 Feld6 - Feld4 == 0 -1.464730.84245 -1.7390.506 Feld6 - Feld5 == 0 0.136050.84245 0.1611.000 (Adjusted p values reported -- single-step method) 2) So my main question is whether I really applied the correct data analysis to the data? Perhaps the discrepancy between the aov-results and the lme-results indicate the need to perform the post-hoc tests correctly. On the other hand, the difference between a p-value of 5.953e-06 (aov) and p0.1 (for all pairwise comparisons) simply indicates, that I did something wrong... Best wishes Thomas [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Post-hoc tests for repeated measures in balanced experimental design
Dear Dennis, thank you for your fast response. Perhaps I should have described the experimental situation in more detail. This tells you that Subject is being treated as a random block factor, and that Conditions 1 and 2 are combinations of treatments applied to each subject. In other words, this describes a randomized block design, so ordering in time is not represented in this model. Moreover, within-subject correlation due to the supposed repeated measures is not represented, either. In each subject we perform one recording for every possible combination of values of both factors Condition1 and Condition2. Ordering in time is not an issue here, because we do not want to study longitudinal changes in time. On the other hand, as each combination of the 2 conditions was applied in each subject within an experimental session, I thought this must be analysed with a repeated-measures approach. As an example, the following lines show the beginning of a typical result file for the first 2 subjects A and B Nr Subject Condition1 Condition2 Result 1 A C1a C2a 5. 2 A C1a C2b 3. 3 A C1b C2a 3. 4 A C1b C2b 3. 5 A C1c C2a 1. 6 A C1c C2b 4. 7 B C1a C2a 5. 8 B C1a C2b 4. 9 B C1b C2a 3. 10 B C1b C2b 2. 11 B C1c C2a 2. 12 B C1c C2b 3. ... If this is a randomized block design, which is the best way to analyse the effect of the two factors Condition1 and Condition2 on Result? How to do post-hoc tests to see whether there significant differences, e. g., between C1a and C1b for Condition1? ano - aov(d$wPatternPulseSNR~ d$Bedingung*d$Felder + Error(d$VPerson/ (d$Bedingung*d$Felder), data=d)) Something seems amiss here, too. Where is the time element represented? Where is the correlation structure on the supposed repeated measures? I also think that you should have Person/Bedingung:Felder as the within- subject error term instead As mentioned above, there is no specific time element, but the repeated measures of the same type of result (here wPatternPulseSNR) under different experimental conditions (here d$Bedingung*d$Felder) should contain the correlation structure. As far as I understood the nomenclature, Bedingung:Felder means a different situation (split-plot) from the situation described above. It's entirely possible that the 'significance' seen in the above tests is due to an incorrect choice of error term. If Bedingung and Felder are both applied within-subject, then the appropriate error term should be the average within-person variance. A repeated measures design is structured rather similarly to a split-plot, except that in a RM design the within-subject observations are correlated in time and therefore the correlation can vary as a function of time - e.g., AR (1), where the correlation between observations decreases exponentially with the time gap between them. (In contrast, observations at the split-plot level are usually assumed to be equi-correlated due to randomization of levels of treatments within subjects.) In your case, there is no 'whole-plot/between subject' treatment - everything is at the within-subject level. ... I don't think you have the right model if this is indeed a repeated measures design, but I'll let others weigh in on the subject. As alluded to above, your model doesn't seem to represent the time element or the correlation structure that one would expect in a repeated measures design. I'd worry about getting the model right before proceeding to the multiple comparisons. Hm, being not an expert in statistics I thank you very much for advice and I agree that the right model is essential. Perhaps my clarification of the data might help to fix this aspect. Everything is at the within-subject level this is exactly true. Thanks again Thomas [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
