The more general concern about significance testing notwithstanding, I have
a question about the use of testing, or other inferential statistical
techniques, in experiments using human subjects, or any other research
method that does not use probability sampling...
Now, all of these tests that we run--whether from ANOVA, regression,
difference of means, correlations, etc.--are based on the assumption that we
have sampled from a population using some probability sampling procedure.
And the meaning of p is inextricably linked to the properties of the
sampling distribution.
However, little experimental research with human subjects is done using a
sample. Most often, in my experience, these studies use volunteer subjects
or naturally existing groups. These subjects are then randomly assigned to
treatment and control groups. Yet, in every instance that I know of,
results are presented with tests of significance. It seems to me that
outside of the context of probability sampling, these tests have no meaning.
Despite this, presentation of such results with tests of significance are
common.
Is there a reasonable interpretation of these results that does not rely on
the assumption of probability sampling? It seems to me that simply
presenting and comparing descriptive results, perhaps mean differences,
betas from a regression, or some other measure of effect size without a test
would be more appropriate. This would however be admitting that results have
no applicability beyond the participants in the study. Moreover, these
results say nothing about the number of subjects one has, which p might help
with regard to establishing minimum believability. Yet, conceptually, p
doesn't work.
Am I missing the boat here? Significance testing in these situations seem to
go over just fine in journals. Appreciate your clarification of the issue.
Regards
Brett
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
