Re: one-way ANOVA question
On 13 Feb 2002 09:48:41 -0800, [EMAIL PROTECTED] (Dennis Roberts) wrote: At 09:21 AM 2/13/02 -0600, Mike Granaas wrote: On Fri, 8 Feb 2002, Thomas Souers wrote: 2) Secondly, are contrasts used primarily as planned comparisons? If so, why? I would second those who've already indicated that planned comparisons are superior in answering theoretical questions and add a couple of comments: another way to think about this issue is: what IF we never had ... nor will in the future ... the overall omnibus F test? would this help us or hurt us in the exploration of the experimental/research questions of primary interest? - not having it available, even abstractly, would HURT, because we would be without that reminder of 'too many hypotheses'. In practice, I *do* consider the number of tests. Just about always. Now, I am not arguing that the particular form of having an ANOVA omnibus-test is essential. Bonferroni correction can do a lot of the same. It just won't always be as efficient. i really don't see ANY case that it would hurt us ... and, i can't really think of cases where doing the overall F test helps us ... But, Dennis, I thought you told us before, you don't appreciate hypothesis testing ... I thought you could not think of cases where doing *any* F-test helps us. [ ... ] -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: one-way ANOVA question
On Fri, 8 Feb 2002, Thomas Souers wrote: 2) Secondly, are contrasts used primarily as planned comparisons? If so, why? I would second those who've already indicated that planned comparisons are superior in answering theoretical questions and add a couple of comments: 1) an omnibus test followed by pairwise comparisons cannot clearly answer theoretical questions involving more than two groups. Trend analysis is one example where planned comparisons can give a relatively unambigious answer (is there a linear, quadratic, etc trend?) where pairwise tests leave the research trying to interpret the substantive meaning of a particular pattern of pairwise differences. 2) planned comparisons require that the researcher think through the theoretical implications of their research efforts prior to collecting data. It is too common for folks to gather some data appropriate for an ANOVA, without thinking through the theoretical implications of their possible results, analyze it with an omnibus test (Ho: all the means the same) and rely on post-hoc pairwise comparisons to understand the theoretical meaning of their findings. In a multi-group design if you cannot think of at least one meaningful contrast code prior to collecting the data, you haven't really thought through your research. 3) your power is better. It is well known that when you toss multiple potential predictors into a multiple regression equation you run the risk of washing out the effect of a single good predictor by combining it with one or more bad predictors. ANOVA is a special case of multiple regression where each df in the between subjects line represents a predictor (contrast code). By combining two or more contrast codes into a single omnibus test you reduce your ability to detect meaningful differences amongst the collection of non-differences. Hope this helps. Michael *** Michael M. Granaas Associate Professor[EMAIL PROTECTED] Department of Psychology University of South Dakota Phone: (605) 677-5295 Vermillion, SD 57069 FAX: (605) 677-6604 *** All views expressed are those of the author and do not necessarily reflect those of the University of South Dakota, or the South Dakota Board of Regents. = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: one-way ANOVA question
Thomas Souers wrote: Hello, I have two questions regarding multiple comparison tests for a one-way ANOVA (fixed effects model). 1) Consider the Protected LSD test, where we first use the F statistic to test the hypothesis of equality of factor level means. Here we have a type I error rate of alpha. If the global F test is significant, we then perform a series of t-tests (pairwise comparisons of factor level means), each at a type I error rate of alpha. This may seem like a stupid question, but how does this test preserve a type I error for the entire experiment? As you (nearly) say, [Only i]f the global F test is significant, we then perform a series of t-tests 2) Secondly, are contrasts used primarily as planned comparisons? If so, why? It depends on the research question. = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: one-way ANOVA question
At 09:21 AM 2/13/02 -0600, Mike Granaas wrote: On Fri, 8 Feb 2002, Thomas Souers wrote: 2) Secondly, are contrasts used primarily as planned comparisons? If so, why? I would second those who've already indicated that planned comparisons are superior in answering theoretical questions and add a couple of comments: another way to think about this issue is: what IF we never had ... nor will in the future ... the overall omnibus F test? would this help us or hurt us in the exploration of the experimental/research questions of primary interest? i really don't see ANY case that it would hurt us ... and, i can't really think of cases where doing the overall F test helps us ... i think mike's point about planning comparisons making us THINK about what is important to explore in a given study ... is really important because, we have gotten lazy when it comes to this ... we take the easy way out of testing all possible paired comparisons when, it MIGHT be that NONE of these are really the crucial things to be examined Dennis Roberts, 208 Cedar Bldg., University Park PA 16802 Emailto: [EMAIL PROTECTED] WWW: http://roberts.ed.psu.edu/users/droberts/drober~1.htm AC 8148632401 = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: one-way ANOVA question
You have to keep in mind that the LSD is concerned with familywise error rate, which is the probability that you will make at least one type I error in your set of conclusions. For the familywise error rate, 3 errors are no worse than 1. Suppose that you have three groups. If the omnibus null is true, the probability of erroneously rejecting the null with the overall Anova is equal to alpha, which I'll assume you set at .05. IF you reject the null, you have already made one type I error, so the chances of making more do not matter to the familywise error rate. Your Type I error rate is .05. Now suppose that the null is false-- mu(1) = mu(2) /= mu(3). Then it is not possible to make a Type I error in the overall F, because the omnibus null is false. There is one chance of making a Type I error in testing individual means, because you could erroneously declare mu(1) /= mu(2). But since the other nulls are false, you can't make an error there. So again, your familywise probability of a Type I error is .05. Now assume 4 means. Here you have a problem. It is possible that mu(1) = mu(2) /= mu(3) = mu(4). You can't make a Type I error on the omnibus test, because that null is false. But you will be allowed to test mu(1) = mu(2), and to test mu(3) = mu(4), and each of those is true. So you have 2 opportunities to make a Type I error, giving you a familywise rate of 2*.05 = .10. So with 2 or 3 means, the max. familywise error rate is .05. With 4 or 5 means it is .10, with 6 or 7 means it is .15, etc. But keep in mind that, at least in psychology, the vast majority of experiments have no more than 5 means, and many have only 3. In that case, the effective max error rate for the LSD is .10 or .05, depending on the number of means. Other the other hand, if you have many means, the situation truly gets out of hand. Dave Howell At 10:37 AM 2/8/2002 -0800, you wrote: Hello, I have two questions regarding multiple comparison tests for a one-way ANOVA (fixed effects model). 1) Consider the Protected LSD test, where we first use the F statistic to test the hypothesis of equality of factor level means. Here we have a type I error rate of alpha. If the global F test is significant, we then perform a series of t-tests (pairwise comparisons of factor level means), each at a type I error rate of alpha. This may seem like a stupid question, but how does this test preserve a type I error for the entire experiment? I understand that with a Bonferroni-type procedure, we can test each pairwise comparison at a certain rate, so that the overall type I error rate of the experiment will be at most a certain level. But with the Protected LSD test, I don't quite see how the comparisons are being protected. Could someone please explain to me the logic behind the LSD test? 2) Secondly, are contrasts used primarily as planned comparisons? If so, why? I would very much appreciate it if someone could take the time to explain this to me. Many thanks. Go Get It! Send FREE Valentine eCards with Lycos Greetings http://greetings.lycos.com = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ = ** David C. Howell Phone: (802) 656-2670 Dept of Psychology Fax: (802) 656-8783 University of Vermont email: [EMAIL PROTECTED] Burlington, VT 05405 http://www.uvm.edu/~dhowell/StatPages/StatHomePage.html http://www.uvm.edu/~dhowell/gradstat/index.html
Re: one-way ANOVA question
At 10:37 AM 2/8/02 -0800, Thomas Souers wrote: 2) Secondly, are contrasts used primarily as planned comparisons? If so, why? well, in the typical rather complex study ... all pairs of possible mean differences (as one example) are NOT equally important to the testing of your theory or notions so, why not set up ahead of time ... THOSE that are (not necessarily restricted to pairs) you then follow ... let the other ones alone no law says that if you had a 3 by 4 by 3 design, that the 3 * 4 * 3 = 36 means all need pairs testing ... in fact, come combinations may not even make a whole lot of sense EVEN if it is easier to work them into your design I would very much appreciate it if someone could take the time to explain this to me. Many thanks. Go Get It! Send FREE Valentine eCards with Lycos Greetings http://greetings.lycos.com = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ = Dennis Roberts, 208 Cedar Bldg., University Park PA 16802 Emailto: [EMAIL PROTECTED] WWW: http://roberts.ed.psu.edu/users/droberts/drober~1.htm AC 8148632401 = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: one-way ANOVA question
Hi On 8 Feb 2002, Thomas Souers wrote: 2) Secondly, are contrasts used primarily as planned comparisons? If so, why? There are a great many possible contrasts even with a relatively small number of means. If you examine the data and then decide what contrasts to do, then you have in some informal sense performed a much larger set of contrasts than you actually formally test. Specifying the contrasts in advance means that you have only performed the number of statistical tests actually calculated. Another (related) way to think of it is that planned contrasts take advantage of pre-existing theory and data to perform tests that favor certain outcomes. To do this, however, contrasts must be specified independently of the data (i.e., planned). Perhaps could be thought of as some kind of quasi-bayesian thinking? That is, given a priori factors favoring certain outcomes, the actual data does not need to be as strong to tilt the results in that direction. Best wishes Jim James M. Clark (204) 786-9757 Department of Psychology(204) 774-4134 Fax University of Winnipeg 4L05D Winnipeg, Manitoba R3B 2E9 [EMAIL PROTECTED] CANADA http://www.uwinnipeg.ca/~clark = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
