Magill, Brett wrote:
> The more general concern about significance testing notwithstanding, I have
> a question about the use of testing, or other inferential statistical
> techniques, in experiments using human subjects, or any other research
> method that does not use probability sampling...
>
> Now, all of these tests that we run--whether from ANOVA, regression,
> difference of means, correlations, etc.--are based on the assumption that we
> have sampled from a population using some probability sampling procedure.
> And the meaning of p is inextricably linked to the properties of the
> sampling distribution.
>
> However, little experimental research with human subjects is done using a
> sample.
maybe a nit pick, but on the contrary, they are _all_ done with a
sample. - a group of measured elements of the set of the population.
The sample must be finite, which it is.
> Most often, in my experience, these studies use volunteer subjects
> or naturally existing groups. These subjects are then randomly assigned to
> treatment and control groups.
If we can assert, loudly, that the subjects are 'representative of the
population' that we care about, then how we obtained them is of no
concern, I'd say. I can, indeed must, define the 'population' to
include the group of non-subjects I wish to make a prediction on. For
example, I would not test females with Viagra, as I don't believe it
will have any effect (I'm guessing in technical ignorance, here). a
counter example: did the first tests with Thalidomide include any
pregnant women? I guess no. Then nobody should predict how the drug
will work on pregnant women (or that it will not have side effects).
The tragedy was that the predictions made (whether explicitly or by
uninformed decision/action) were proven absolutely false.
the difficulty, as I see it, is that occasionally, we define a
population, solicit some subjects, and then assert that the subjects
match the population, when they don't. the solicitation of volunteers
occasionally has a selection effect. For example, in the USA it tends
to under represent African Americans, who often distrust medical
researchers.
> Yet, in every instance that I know of,
> results are presented with tests of significance. It seems to me that
> outside of the context of probability sampling, these tests have no meaning.
> Despite this, presentation of such results with tests of significance are
> common.
>
> Is there a reasonable interpretation of these results that does not rely on
> the assumption of probability sampling? It seems to me that simply
> presenting and comparing descriptive results, perhaps mean differences,
> betas from a regression, or some other measure of effect size without a test
> would be more appropriate. This would however be admitting that results have
> no applicability beyond the participants in the study. Moreover, these
> results say nothing about the number of subjects one has, which p might help
> with regard to establishing minimum believability. Yet, conceptually, p
> doesn't work.
>
> Am I missing the boat here? Significance testing in these situations seem to
> go over just fine in journals. Appreciate your clarification of the issue.
>
> Regards
>
> Brett
How would you design a study, using a Baysian approach?
Jay
--
Jay Warner
Principal Scientist
Warner Consulting, Inc.
4444 North Green Bay Road
Racine, WI 53404-1216
USA
Ph: (262) 634-9100
FAX: (262) 681-1133
email: [EMAIL PROTECTED]
web: http://www.a2q.com
The A2Q Method (tm) -- What do you want to improve today?
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
