OK, clear enough. I assume by ANOVA you mean an analysis of the 2x3 table in your example, and by regression you mean the analysis of the same data, but using each measured value from the "mental rotation" test. (ANOVA and regression are the same when the regression uses the 3 levels as a variable.)
The problem is not "random" assignment, but the fact that the mental rotation variable is a variable containing error. The model is y=a+(b+e1)+e2 and not y=a+b+e, as would be correct for ANOVA or regression. Using the model y=a+b+e produces inconsistent estimators and erroneous statistical tests. This is theory. In practice, the conclusions are not always bad, but will be so in the example you give. If the second variable is a test like mental rotation, the errors are likely to be serious because the 3 levels are not widely separated. If it were possible to replicate the test, the values for the same individual would be distributed over all three levels -- hopefully, with heavier concentration on one level. If you must analyze y=a+b+e, then you should not claim significance at the usual 5% level, but insist on a significance of 1% or even less. The proper treatment of this problem is total least squares, TLS. It is not a difficult calculation, but I don't know of a package that does it: but that is my ignorance of packages. I'm sure someone in the group will know and may offer a suggestion. A book on the subject is Van Huffel, S., and Vanderwalle, J. (1991). The total least squares problem. SIAM. It will probably be heavy going for you. Steve wrote: > Thanks for your replies thus far. > > I don't have a specific problem per se. I'm trying to better understand the > GLM and the limitations of ANOVA. > > So, in a nutshell, here's my question: > > IF a participant is NOT randomly assigned to a level of a factor BECAUSE the > factor in question CANNOT be randomly assigned (i.e. gender, IQ, etc.), WHAT > are the mathematical/statistical consequences. > > The domain of interest is Psychology/Behavioral Sciences. The problem is > that many researchers will examine data for group mean differences across > factors that CANNOT be randomly assigned. This seems to violate a basic > assumption of ANOVA- random assignment to treatment groups. > > > > EXAMPLE (purely bogus- I'm making this up on the fly): > > A researcher wants to determine whether or not the use of pictures in an > instruction manual has an impact on how well participants can assemble the > device depicted in the instructions. Two instruction manuals are created: > one with pictures and one without (text only). The researcher believes that > the picture instruction manuals will work best for those participants who > score higher on a "mental rotation ability" test. So, participants are given > the mental rotation ability test and are sorted into one of three groups: > low, medium, and high mental rotation ability. The result is a 2x3 BS > design. > > > Notice that participants were randomly assigned to only one of the two > factors- instruction type. "Assignment" to the other factor was not at all > random; instead, participants were tested on their mental rotation ability > and sorted into one of three groups according to ability. > > I would personally use Multiple Regression to tackle this problem (maybe > that's not a reasonable approach?). BUT, I KNOW for a fact that many > behavioral scientists WOULD use ANOVA in this type of situation. > > Again, I am NOT working on a project with this type of data; No one I know > is doing the research described above. I would NOT use ANOVA in the above > situation. I HONESTLY want to understand the proper use of ANOVA. > > TWO QUESTIONS: > > 1) Is there a problem with using ANOVA to analyze data when random > assignment was NOT used for one or more of the factors? > > 2) IF YES, why? > > 3) IF YES, what are the implications (i.e. Type I error inflation, wrong SS > values, etc.)? > > > Sorry for the confusion that I have caused. I know enough about stats to be > dangerous. > Thanks > Steve > > -- Bob Wheeler --- http://www.bobwheeler.com/ ECHIP, Inc. --- Randomness comes in bunches. . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
