Hi Dennis,
Robert's observation is 'spot on' because it is the way things are, rather than the
way we would like to think things are. I (of course) agree that people writing
papers should have some sense of proportion the claims made in their papers.
Nevertheless, if you want to study the gag reflex in humans, you simply cannot take
a simple random sample of all humans, so you have to use a surrogate population of
some sort.
In fact I would claim that except for artificial (ie class room) examples, you
pretty well always have to use surrogate populations. This is of course
particularly true when the population is not well defined. In market research, for
example, when your population allegedly consists of 'customers of XYZ Store', but
equally well in pretty well any branch of research.
It has been my opinion for quite some time now that the uncertainties in
conclusions due to use of surrogate populations, plus those due to measurement (eg
uncertainty in interpretation of questions in a questionnaire), far exceed sampling
errors - probably even exceed nonsampling errors.
However, this is quite consistent with my observations about models. In applying
the results obtained from the surrogate population to the general 'true' population
of interest, we apply those results as a model. If the surrogate population was
well chosen (and the analysis well done) then the model is likely to be reasonably
appropriate. That is, it is likely to 'work'.
It cannot be emphasized too much that the statistical analysis - including the
definition of variables, the design, the collection of data and the analysis of the
data - is simply a part of a process of investigation. This part provides evidence
- to some extent objective, and hopefully objective - which will help the
researcher to argue his or her case. It is only part of the evidence. And that
evidence may be sufficiently strong to persuade the scientific community that the
researcher's argument is valid, or it may not.
Regards,
Alan
dennis roberts wrote:
> >Robert Dawson wrote:
> >
> >> As far as random samples are concerned: it is *very* rare for a true
> >> random sample, based on an equal-probability sample of the population to
> >> which the inference is intended to extend, to be taken. Say a researcher is
> >> studying the behaviour of humans. (S)he may take a random sample from the
> >> student subject pool, but not from the human race; and yet the paper
> >> published will claim to be about "Artificially Inducing The Gag Reflex in
> >> Humans", not "Artificially Inducing The Gag Reflex in Students Enrolled in
> >> Psych 1000 at Miskatonic U. (Fall '00)".
>
> well, perhaps journal editors should INSIST that the author say very
> clearly ... that this only applies to students enrolled in psy 1000 at
> miskatonic u ... fall 1999 ... since that is what it is ...
>
> the only way we can get around this ... is to REPLICATE investigations and
> see if we can find comparable results across disparate subject pools ..
> but, unfortunately ... if you do like benton j underwood did many years
> ago: studies in the meaningfulness of learning 1 ... then 2 ... then ... 29
> ... your tenure would be 'on hold' ... you are not allowed to replicate 10
> times ... you must move onward and upward ...
>
> we would be MUCH better off ... reducing drastically the NUMBER of things
> we tried to be unique in (that no one else has done) ... and spend more
> time replicating work ... that is deemed to be MORE important ... in the
> long run ... our knowledge base would be better and stronger ... rather
> than relying on some p value to suggest that THIS has been researched and
> THE answer found ... now we should move on to something else ... just
> another weight place on the poor little p ... when its back is already
> crushing!
>
> the more i think about it ... the more i think our overall effort is
> misguided ... and this is but one reason why there is so much crappy
> research ... and believe me ... there is plenty to go around across all the
> disciplines
>
> Even if some future world
> >> government were to allow researchers access to a list of all humans alive at
> >> some moment to use as a sampling frame, most researchers would not disclaim
> >> any applicability of their research to those dead or not yet born. The
> >> implicit "Platonic" population larger than that available for study is a
> >> problem that is always with us; a bad sample is one in which this causes
> >> bias. The situation in which the entire actual population is available for
> >> study is an extreme case, of course.
>
> i would suggest that inferential statistics .. as we know it ... is not
> robust to cruddy samples and if samples are cruddy ... what's the point in
> using some standard error that is BASED on the assumption that samples are
> NOT cruddy ... but rather, have some connection to random error ...
>
> you can't have it both ways ... either we make a good faith effort to
> sample in a reasonble way .. such that our standard errors can be expected
> to be about right ... or, we say that samples won't be good ... and if that
> is true ... forget the notion of using the standard error in some rigid way
> for making hypothesis tests ... confidence intervals ... and the like
>
> when we use such error estimates as: stan error of the mean = S / sqrt n ..
> does this apply no matter what?
>
> there is a daisy chain here ... the hypothesis is about a population ...
> and, we use the data from our sample to make a decision about that
> hypothesized parameter ... BUT, if our sample cannot be considered (within
> some fudge factor) to be representative of the population to which we have
> made this hypothetical stab ... seems like we need to pack it in
>
> let's say that we take a sample by any means ... and, the question we have
> formulated is that .. in the population ... we will find some 6's ... AND,
> we happen to find 1 or more 6's ... now, i don't care how you took the
> sample ... good way or bad way ... we have confirmed our question ... but,
> what if you don't find any 6's??? i would say in this case ... you are up
> the creek ... since there is no model we can apply given we know nothing
> about how the sample was taken ...
>
> if we have to assume that samples can be anything ... since we can never
> EXACTLY get a truly random sample ... then we are in a peck of trouble ...
>
> i recall a number of posts that alan made ... arguing rather vehemently
> about the fact that we need a model for our data ... well, what is the
> model for our data if we have no control over our sampling ... nor any way
> to have a crack at estimating the error BASED on that sample information?
> but now ... in telling robert ... spot on ... in the context that robert is
> implying that it is ok to go ahead and make these inferences even if our
> sampling methods are poor ... so, the way i read this ... alan is more or
> less agreeing with that .. and that does not appear to be a very consistent
> approach to things ...
>
> bottom line: how goes your samples ... that's where your inferences are headed
>
> but that is just my read of it
> >>
> >> -Robert Dawson
> >>
> >> ===========================================================================
> >> This list is open to everyone. Occasionally, less thoughtful
> >> people send inappropriate messages. Please DO NOT COMPLAIN TO
> >> THE POSTMASTER about these messages because the postmaster has no
> >> way of controlling them, and excessive complaints will result in
> >> termination of the list.
> >>
> >> For information about this list, including information about the
> >> problem of inappropriate messages and information about how to
> >> unsubscribe, please see the web page at
> >> http://jse.stat.ncsu.edu/
> >> ===========================================================================
> >
> >--
> >Alan McLean ([EMAIL PROTECTED])
> >Department of Econometrics and Business Statistics
> >Monash University, Caulfield Campus, Melbourne
> >Tel: +61 03 9903 2102 Fax: +61 03 9903 2007
> >
> >
> >
> >
> >===========================================================================
> >This list is open to everyone. Occasionally, less thoughtful
> >people send inappropriate messages. Please DO NOT COMPLAIN TO
> >THE POSTMASTER about these messages because the postmaster has no
> >way of controlling them, and excessive complaints will result in
> >termination of the list.
> >
> >For information about this list, including information about the
> >problem of inappropriate messages and information about how to
> >unsubscribe, please see the web page at
> >http://jse.stat.ncsu.edu/
> >===========================================================================
>
> ==============================================================
> dennis roberts, penn state university
> educational psychology, 8148632401
> http://roberts.ed.psu.edu/users/droberts/droberts.htm
>
> ===========================================================================
> This list is open to everyone. Occasionally, less thoughtful
> people send inappropriate messages. Please DO NOT COMPLAIN TO
> THE POSTMASTER about these messages because the postmaster has no
> way of controlling them, and excessive complaints will result in
> termination of the list.
>
> For information about this list, including information about the
> problem of inappropriate messages and information about how to
> unsubscribe, please see the web page at
> http://jse.stat.ncsu.edu/
> ===========================================================================
--
Alan McLean ([EMAIL PROTECTED])
Department of Econometrics and Business Statistics
Monash University, Caulfield Campus, Melbourne
Tel: +61 03 9903 2102 Fax: +61 03 9903 2007
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
