Re: What's type III?

[Donald Burrill]

On Mon, 20 Nov 2000, Karl L. Wuensch wrote:

> Chris said :"Since both the null and alternative are generally false,"
> 
> Now I'm confused.  I always thought that null and alternative were 
> mutually exclusive and exhaustive, as in "parameter LE value" versus 
> "parameter GT value."

No, you're not confused.  Chris is.  ;-)
-- Don.
 --
 Donald F. Burrill[EMAIL PROTECTED]
 348 Hyde Hall, Plymouth State College,  [EMAIL PROTECTED]
 MSC #29, Plymouth, NH 03264 (603) 535-2597
 Department of Mathematics, Boston University[EMAIL PROTECTED]
 111 Cummington Street, room 261, Boston, MA 02215   (617) 353-5288
 184 Nashua Road, Bedford, NH 03110  (603) 471-7128



=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Karl L. Wuensch]

Chris said :"Since both the null and alternative are generally false,"

Now I'm confused.  I always thought that null and alternative were mutually
exclusive and exhaustive, as in "parameter LE value" versus "parameter GT
value."

++ Karl L. Wuensch, Department of Psychology, East Carolina University,
Greenville NC 27858-4353 Voice: 252-328-4102 Fax: 252-328-6283
[EMAIL PROTECTED] http://core.ecu.edu/psyc/wuenschk/klw.htm



=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Christopher Auld]

Rich Ulrich  <[EMAIL PROTECTED]> wrote:

>Well, I can squint my eyes, and posit a Type III error  that meets
>that definition.  But it's virtually never going to happen, or be
>regarded as an event-of-that-class when it does; so it is not "on a
>par with" the other two.  In my opinion.

It seems to me the concept of "Type III error" discussed implicitly
rests on the notion that rejecting the null implies accepting the
alternative.  Since both the null and alternative are generally false,
I don't see how we can square the concept itself with the underpinnings
of frequentist inference.

-- 
Chris Auld  (403)220-4098
Economics, University of Calgary
Calgary, Alberta, Canadahttp://jerry.ss.ucalgary.ca/>


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Rich Ulrich]

On 16 Nov 2000 17:18:44 GMT, [EMAIL PROTECTED] (Radford Neal)
wrote:

> In article <[EMAIL PROTECTED]>,
> Rich Ulrich  <[EMAIL PROTECTED]> wrote:
> 
> > There is another definition [ of "Type III error" ] cited a few
> > times which is seemingly technical, "rejecting the null, but in the
> > wrong direction".  I think that is a similar sneer at bone-headedness.  
> > There is no "wrong direction" with a two-tailed test...
> 
> Are you really suggesting that researchers who reject H0: mu=0 with a
> two-tailed test act thereafter as if they know that mu is not zero,
> but have no idea whether it's positive or negative?  I think this is
> highly implausible.  Such "Type III" errors seems to me to be quite on
> a par with Type I and Type II errors.
> 

Well, I can squint my eyes, and posit a Type III error  that meets
that definition.  But it's virtually never going to happen, or be
regarded as an event-of-that-class when it does; so it is not "on a
par with" the other two.  In my opinion.

For that sort of Type III:
Here is a calcium-something-beta-blocker, for preventing some version
of heart disease,  that was backed by a tiny amount of data analysis,
on top of a bunch of reasonable-sounding analogies.  When the new
study came in, saying XX was bad instead of good, the media treated it
as an old, bad assumption, now (to some surprise) overturned.  They
say, the old XX was never "well supported" in the first place.

I don't see how that gains coherence by ringing in "Type III."   

If the old one was "well supported" after all, then we are not
talking about using the first really-good data to replace a minor
statistical artifact, as we were before.   If it had been
well-supported, the new overthrow of XX did require some change in
QUESTION:  or, so I presume.

I think that I could get lost here, without some examples of what
someone has in mind.

-- 
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Radford Neal]

In article <[EMAIL PROTECTED]>,
Rich Ulrich  <[EMAIL PROTECTED]> wrote:

> There is another definition [ of "Type III error" ] cited a few
> times which is seemingly technical, "rejecting the null, but in the
> wrong direction".  I think that is a similar sneer at bone-headedness.  
> There is no "wrong direction" with a two-tailed test...

Are you really suggesting that researchers who reject H0: mu=0 with a
two-tailed test act thereafter as if they know that mu is not zero,
but have no idea whether it's positive or negative?  I think this is
highly implausible.  Such "Type III" errors seems to me to be quite on
a par with Type I and Type II errors.

   Radford Neal


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Rich Ulrich]

On Thu, 16 Nov 2000 08:40:38 GMT, Kresten <[EMAIL PROTECTED]> wrote:
> 
> > Howard Raiffa (Decision Analysis, footnote, p. 264) agrees that
> > errors
> > of the third kind are "solving the wrong problem", and attributes
> > this to John Tukey
> 
> My ref is:
> 
> Kimball, AW (1957)
> Errors of the third kind in statistical consulting
> J Am Stat Assoc 57, 133
> 
> Haven't got the paper, though.

We seemed to have established "Kimball" as the originator, a couple of
weeks ago.  In my post of Oct 31, I reported that Google finds various
citations, with Kimball being the most frequent (and earliest).  

Also on the Web, there are references to Dobson and Cook, whose Type
III and Type IV have some currency in evaluation research -- These
seem to be an extension of the wisecrack, making it particular to
their area.
> > "... Evaluators commonly make two types of errors doing evaluations: 
Type III error is measuring something that does not exist; 
Type IV error is measuring something that is of no interest to
management and policy maker." (Scanlon et al., 1977,p.36 , cit. after
Dobson & Cook, 1980, p. 270 ).


There is another definition cited a few times which is seemingly
technical,
"rejecting the null, but in the wrong direction".  I think that is a
similar sneer at bone-headedness.  There is no "wrong direction" with
a two-tailed test, and, again, it implies for the one-tailed test that
you are "asking the wrong question."   As someone else posted more
elegantly, it surely is not on the same plane as the technical
statements of "Type I" and "Type II" error.

-- 
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Kresten]

> Howard Raiffa (Decision Analysis, footnote, p. 264) agrees that
> errors
> of the third kind are "solving the wrong problem", and attributes
> this to John Tukey

My ref is:

Kimball, AW (1957)
Errors of the third kind in statistical consulting
J Am Stat Assoc 57, 133

Haven't got the paper, though.

Kresten


Sent via Deja.com http://www.deja.com/
Before you buy.


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Bill Jefferys]

In article <8uttqd$v15$[EMAIL PROTECTED]>, Kresten <[EMAIL PROTECTED]> 
wrote:

#> >What is a type III statistical error?

Howard Raiffa (Decision Analysis, footnote, p. 264) agrees that errors 
of the third kind are "solving the wrong problem", and attributes this 
to John Tukey. He also nominates for error of the fourth kind "solving 
the right problem too late".

Bill

-- 
Bill Jefferys/Department of Astronomy/University of Texas/Austin, TX 78712
Email: replace 'warthog' with 'clyde' | Homepage: quasar.as.utexas.edu
I report spammers to [EMAIL PROTECTED]
Finger for PGP Key: F7 11 FB 82 C6 21 D8 95  2E BD F7 6E 99 89 E1 82
Unlawful to use this email address for unsolicited ads: USC Title 47 Sec 227


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Kresten]

> >What is a type III statistical error?

As far as I recall:

In the book:
Statistics for experimenters : an introduction to design, data
analysis, and model building
by Box, Hunter^2

there is a reference (I can find it at home) to another stat-book in
which typeIII is defined as:

"getting the right answer to the wrong problem"

HTH
Kresten


Sent via Deja.com http://www.deja.com/
Before you buy.


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Robert J. MacG. Dawson]

"Werner W. Wittmann" wrote:
> 
> > Herman and N.N.,
> > type III error means measuring the wrong construct or something
> > nonexistent.In my German book about evaluation research(1985) I cited the
> > following:
> > "Statistician worry about two types of errors..:
> > Type I error is rejecting a hypothesis when it should be accepted; Type II
> > error is accepting a hypothesis when it should be rejected ...
> > Evaluators commonly make two types of errors doing evaluations: Type III
> > error is measuring something that does not exist; Type IV error is
> measuring
> > something that is of no interest to management and policy maker."

I always worry about this when it gets beyond being an "in-joke" like
the Eleventh Commandment ("Thou shalt not get found out.") and reaches a
wider audience, because of the pedagogical risks of conflating two sorts
of "error".  The terms "Type I error" and "Type II error" do seem to
lead novices to think of them as errors in the sense of something that
wouldn't have happened if you'd been more careful or sensible; and
lumping them with genuine errors in this sense  compounds the confusion.

The fault is perhaps with the original terminology, not the joke. The
terms "Type N error" for N=I to II are a truly dreadful example of bad
coinage; they are practically impossible to define directly and almost
everybody who tries gets it wrong.  

A Type I error is NOT 'rejecting a hypothesis when it should be
accepted'. A hypothesis test is a (byzantine and often wrong-headed, but
let that pass) process that converts data to decisions. If the data
point to rejection - even if the data happen to be atypical of the
population they represent - then the hypothesis "should" be rejected. If
you knew the population parameter ahead of time yu wouldn't have done
the test!

Indirect definitions such as "in such a case we shall say that a Type I
error has occurred" are less likely to be utterly wrong, but are not
much more satisfactory as a definition of "Type I error" than "When I
watch the news I tell myself that the world is going to the dogs"" is
satisfactory as a definition of "dog". Yes, the second usage is an idiom
that cannot be understood at the single-word level... so, essentially,
is the first.

In fact, the term "a Type I error" does not have a syntactically
self-contained definition at all. It is a "macro" that cuts across the
syntax of natural language and of probability, and its real meaning is
something like

rejecting a hypothesis, conditional upon its being true

which is only meaningful when you plug it into 

"the probability of (-)"

So you have something that appears syntactically to be an unconditional
probability, and which is, in fact, when "expanded", conditional upon a
certain parameter value!

This is like the sort of abuse of the C preprocessor that some
self-taught programmers revel in in which (for instance) you define "k"
to be "j++". Not only does the fake variable "k" yield a value while
apparently never being initialized, but it invisibly changes its value
(and that of j) every time it's used, so that for instance the equality
test "k == k" returns 0 (false)!   Of course, programmers who insist on
doing things like this are unwelcome on teams trying to write
maintainable code. 

In the case of Type I errors, it is not uncommon for students to take
"the probability of a Type I error" to be P( H_0 is rejected and mu =
mu_0) rather than P(H_0 is rejected | mu=mu_0). The fact that the first
probability is 0 (usually, to a bayesian) or undefined (to a
frequentist) does not stop them. Perhaps one of the reasons is that (
H_0 is rejected and mu = mu_0) at least _sounds_ like something you
could find the probability of.

So maybe we should redefine the first two, and let a Type I error be
"rejecting a null hypothesis instead of estimating effect size" while a
Type II error can become "failing to reject a null hypothesis and
claiming you have shown theta = theta_0" ?

-Robert Dawson


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[William P. Clay]

Herman Rubin wrote:

> In article <8tl9ir$j9g$[EMAIL PROTECTED]>, kj0  <[EMAIL PROTECTED]> wrote:
>
> >What is a type III statistical error?
> >(I know about types I and II).
> >Thanks,
>
> This is the most common type; doing the wrong problem.
>

Herman,

Great one!  Made my day.

--
Bill Clay




=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Karl L. Wuensch]

Although I do not have the source, I have heard Type III error described as
rejecting the null and concluding that there is an effect in one direction,
when, in fact, the effect is, in the population, in the opposite direction.
Assuming that a nondirectional test was employed, the null is false, thus
this error is not a Type I error.

++ Karl L. Wuensch, Department of Psychology, East Carolina University,
Greenville NC 27858-4353 Voice: 252-328-4102 Fax: 252-328-6283
[EMAIL PROTECTED] http://core.ecu.edu/psyc/wuenschk/klw.htm
- Original Message -
kj0  <[EMAIL PROTECTED]> wrote:
>
>
> >What is a type III statistical error?
> >(I know about types I and II).
> >Thanks,
>




=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Rich Ulrich]

On 31 Oct 2000 11:24:44 -0800,
[EMAIL PROTECTED] (Werner W. Wittmann) wrote:

> > Herman and N.N.,
> > type III error means measuring the wrong construct or something
> > nonexistent. 

> > In my German book about evaluation research(1985) I cited the
> > following:
> > "Statistician worry about two types of errors..:
> > Type I error is rejecting a hypothesis when it should be accepted; Type II
> > error is accepting a hypothesis when it should be rejected ...
> > Evaluators commonly make two types of errors doing evaluations: Type III
> > error is measuring something that does not exist; Type IV error is
> measuring
> > something that is of no interest to management and policy maker."
> > (Scanlon et al., 1977,p.36 , cit. after Dobson & Cook, 1980, p. 270 )

 < snip, more citations >

For wittiness, I thought that Herman said it better, 

> > This is the most common type; doing the wrong problem.

Or, better yet, Kimball, 1957:  "Right answer to the wrong question."
Basically, it is a wise-crack.  

So far as I know, Dobson & Cook (cited, above, in the 1970s and 1980s)
are not known to statisticians generally, for  "Type III"  and IV.
But, who knows? if they become standard among Evaluators, maybe their
consistency will spread.

Using Google, Kimball's is what I found most.

-- 
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

RE: What's type III?

[Werner W. Wittmann]

> Herman and N.N.,
> type III error means measuring the wrong construct or something
> nonexistent.In my German book about evaluation research(1985) I cited the
> following:
> "Statistician worry about two types of errors..:
> Type I error is rejecting a hypothesis when it should be accepted; Type II
> error is accepting a hypothesis when it should be rejected ...
> Evaluators commonly make two types of errors doing evaluations: Type III
> error is measuring something that does not exist; Type IV error is
measuring
> something that is of no interest to management and policy maker."
> (Scanlon et al., 1977,p.36 , cit. after Dobson & Cook, 1980, p. 270 )
> My ref. list tells me the following sources:
>
> Scanlon,J.E. et al., Evaluability assessment. Avoiding Type III or IV
> errors. In G.R. Gilbert & P.J. Conklin (Eds.), Evaluation management: A
> sourcebook of readings. Charlottesville,VA: US Civil Service Commission,
> 1977
>
> Dobson,L.D., & Cook, T. J. Avoiding type III error in program
> evaluation.Results from a field experiment.
> Evaluation and Program Planning, 1980,3,269-276.
>
> Cook,T. J. & Dobson, L. D. Reaction to reexamination: More on type III
error
> in program evaluation.
> Evaluation and Program Planning, 1982, 5 , 119-121.
>
> The problems discussed are related to a failed field experiment which is
my
> favorite example in teaching to my students what happens if no real
> manipulation checks are done for the treatment dummy. Nice story and
> discussions
> related to the so-called " challenge experiment" .See:
>
> Cook,T.J. , Dobson,L.D., & Rezmovic,E.L. Working with ex-offenders: The
> challenge experiment.
> Research Triangle Park,NC : Research Triangle Institute, 1980.
>
> HTH
>
> Werner
>
> Werner W. Wittmann; University of Mannheim; Germany;
> e-mail: [EMAIL PROTECTED]
>
>
> - Original Message -
> From: Herman Rubin <[EMAIL PROTECTED]>
> To: <[EMAIL PROTECTED]>
> Sent: Tuesday, October 31, 2000 2:05 PM
> Subject: Re: What's type III?
>
>
> > In article <8tl9ir$j9g$[EMAIL PROTECTED]>, kj0  <[EMAIL PROTECTED]>
wrote:
> >
> >
> > >What is a type III statistical error?
> > >(I know about types I and II).
> > >Thanks,
> >
> > This is the most common type; doing the wrong problem.
> >
> >
> >
> > --
> > This address is for information only.  I do not claim that these views
> > are those of the Statistics Department or of Purdue University.
> > Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette
> IN47907-1399
> > [EMAIL PROTECTED] Phone: (765)494-6054   FAX: (765)494-0558
> >
> >
> > =
> > Instructions for joining and leaving this list and remarks about
> > the problem of INAPPROPRIATE MESSAGES are available at
> >   http://jse.stat.ncsu.edu/
> > =
>



=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=

Re: What's type III?

[Herman Rubin]

In article <8tl9ir$j9g$[EMAIL PROTECTED]>, kj0  <[EMAIL PROTECTED]> wrote:


>What is a type III statistical error?
>(I know about types I and II).
>Thanks,

This is the most common type; doing the wrong problem.



-- 
This address is for information only.  I do not claim that these views
are those of the Statistics Department or of Purdue University.
Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399
[EMAIL PROTECTED] Phone: (765)494-6054   FAX: (765)494-0558


=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
  http://jse.stat.ncsu.edu/
=