Re: Evaluation of skating
Next question: How much does Rasch analysis depend upon the evaluators being ignorant that the method will be used? In other words, can (A) one Rasch-aware judge (B) a minority of Rasch-aware judges (C) a majority of Rasch-aware judges (but not the whole panel) still (a) bring about a desired result (b) without giving scores stranger than are now given? (c) without needing a specially-programmed Palm Pilot or other "smoking gun" to hack the math? ` -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Question on random number generator
Herman Rubin wrote: > > > ExpVar = -ln(UnifVar); > > It is not a good method in the tails, and is much too slow. If I recall correctly, transcendental operations on a Pentium require only a couple hundred clock cycles and can usually be optimized to take place during other calculations; so a few million simulations per second ought to be possible on the average domestic machine. -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Question on random number generator
Alan Miller wrote (six times): > > Linda wrote in message <[EMAIL PROTECTED]>... > >I want to generate a series of random variables, X with exponential > >PDF with a given mean,MU value. However, I only want X to be in some > >specified lower and upper limit?? Say between 0 -> 150 i.e. rejected > >anything outside this range Does anyone have any ideas how should I do > >that?? > > > >Regards, > >Linda > > Is MU the mean before truncation? - or afterwards? > The ziggurat algorithm seems to be the fastest for generating > exponentially-distributed RV's. > You can then simply scale them, by multiplying by the mean BEFORE > truncation, and then throw away any which exceed the upper bound. > Alternatively, following Herman Rubin's idea, you can post the same message to EDSTAT-L repeatedly and let X be the delay until somebody points this out. This should be geometrically distributed, which will approximate the desired exponential distribution For most purposes I do not share Herman's concern about the tails of the distribution. If we use (say) a 64-bit integer as the basis of the uniform distribution, granularity will only be significant for the last few dozen values, which will turn up once every quintillion or so runs. Moreover, fast hardware logarithms are almost a given today. However, his gimmick of randomizing the mantissa and characteristic separately is a good one and well worth remembering. If I recall correctly, math coprocessors use binary logs too, so a super-fast algorithm for (say) a Pentium would probably tie the two approaches together. -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Question on random number generator
Linda wrote: > > I want to generate a series of random variables, X with exponential > PDF with a given mean,MU value. However, I only want X to be in some > specified lower and upper limit?? Say between 0 -> 150 i.e. rejected > anything outside this range Does anyone have any ideas how should I do > that?? For untruncated exponential RV's the negative-log method of converting a uniform [0,1] RV is about as good as you can get: ExpVar = -ln(UnifVar); It can easily be adjusted to censor to any interval [a,b] by prescaling onto [exp(-b),exp(-a)]; TruncExpVar = -ln(exp(-b) + (exp(-a)-exp(-b))*UnifVar); -R. Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Newbie question
AP wrote: > > Hi all: > > I would appreciate your help in solving this question. > > calculate the standard deviation of a sample where the mean and > standard deviation from the process are provided? > E.g. Process mean = 150; standard deviation = 20. What is the SD for > a sample of 25? The answer suggested is 4.0 Right answer, wrong question... You were, almost certainly, not asked for the standard deviation of the sample, but for the standard deviation of the MEAN of the sample. The thing you need to note here is that the sample is obtained through a random process, so that most things computed from the sample are likewise randomized through the sampling process. It is often helpful to think of taking a lot of samples all of the same size, computing the mean (or whatever) for each of them, and then analyzing that set of numbers. In particular, you can calculate the standard deviation. Probability theory tells us that in the population of ALL samples of size N from a population with mean mu and standard deviation sigma, the sample means will have mean mu and standard deviation sigma/sqrt(N). Moreover, as N gets larger, the "sampling distribution" gets closer to a normal distribution, which under some circumstances lets us say more about the distribution based on mu and sigma/sqrt(N). -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: RE :
J. Random Loser in Dnepropetrowsk wrote: > > The "Listsoft & Co" company offers save your money. > We prepositionals the softwere. Ah. That really fills me with confidence. The are : > > 1. MS WINDOWS 2000 PROFESSIONAL + (SERVICE PACK 2)- 1CD -$15 > > At purchase more than 5 CD - AntiViral Toolkit Pro by Eugene > Kaspersky - is free. > > If this latter contradicts yours moral principles - remove this latter Ah, such high principles. I guess he figures that his customers might figure they _deserve_ viruses as a sort of high-tech hairshirt, in penanace for the piracy that they haven't the moral fiber to abstain from. What would Dostoevsky have made of it all? Of course, as fans of Tom Lehrer will recall, Dnepropetrowsk has been an important link in the trade in pirated information for over a century... along with Alexandrowsk, and Omsk, and Tomsk, and Minsk, and Pinsk... -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: area under the curve
> Melady Preece wrote: > > A student wants to know how one can calculate the area under the curve > for skewed distributions. Can someone give me an answer about when a > distribution is too skewed to use the z table? You can only use the z table directly to find the area under a curve when the distribution is a standard normal distribution - that is, when it has precisely the probability density function f(x) = 1/sqrt(2 pi) exp(-x^2/2) You can use it indirectly if the distribution can be transformed into a standard normal distribution. The main cases are: general normal distribution: subtract mu and divide by sigma lognormal distribution: take logs You can use it to *approximate* the area under the curve for near-normal distributions such as binomial (N large), t (nu large), gamma (beta large), beta (both parameters large), or the sampling distribution of the mean from any population that is not too heavy-tailed (N large). "Large" in every case depends on parameter values and desired degree of approximation. To calculate the area under the curve for a skewed distribution you can: (a) integrate the PDF for the distribution (gamma, Weibull, etc) (b) look it up in a table (chi-squared, noncentral t etc) (c) estimate based on the percentiles of a sample. Hope this helps. Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: The efficacy of prayer
Chris wrote: > > I don't know how many of you already noticed the rather humorous story on > the "Effect of remote, retroactive intercessory prayer on..." in the always > somewhat tongue-in-cheek last-of-the-year-issue of the BMJ. > If not, you may find it worth having a look at. > http://bmj.com/cgi/content/full/323/7327/1450 > > The appropriate - though also very tongue-in-cheek - reaction would be to > compare Leibovici's result with those of Francis Galton, as described in his > famous article: "Statistical inquiries into the efficay of prayer", a > reference nevertheless conspiciously absent in Leibovici's ref-list. > http://www.mugu.com/galton/essays/efficacy-prayer.htm > > Apart from the tongue-in-cheek aspects, are there any serious statistical > lessons to be taken from Leibovici's article? > To me at least, it once again clearly demonstrates it's possible to find a > significant difference between two halves of a sample, even when the sample > has been split in two halves by a total random criterion - and isn't that > exactly what classical hypothesis-testing implies? The real lesson here is probably that N>30 does not justify the assumption that the sampling distribution is normal. >From the article: Min Q1 Med Q3 Max Days in hospital Intvn 0 4 7 13 165 Ctrl0 4 8 16 320 Duration of fever Intvn 0 1 2 4 49 Ctrl0 1 2 5 50 These data are enormously skewed; a few data at the high end of each group thus presumably dominate the mean. The fortuitouos inclusion of a few more of these patients in one or the other group would thus create differences in the mean that would be highly improbable in data from equivalent normally-distributed populations. -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: I Hack Into Your Paypal Account!
[EMAIL PROTECTED] wrote: > > Hello, > > My name Steven Lee. I am a programmer and the best. > I can hack into any Paypal order form and get > membership for free. Is a Paypal order form something like Opus Dei? If you no believe me see my > site http://myfriend4ever.com/freehit/index.cgi (illiterate demand for Danegeld omitted) Anyhow, if this loser was daft enough to put his real address on his extortion note, I presume the Hong Kong cops will by now have his computer, codes, person, etc. in their custody... this kiddy makes the Nigerian Widow scammers look positively intellectual (and literate). -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: transformation of dependent variable in regression
"Case, Brad" wrote: > > > Hello. I am hoping that my question can be answered by a statistical > > expert out there!! (which I am not). I am carrying out a multiple linear > > regression with two independents. It seems that a square root > > transformation of the dependent variable effectively decreases > > heterocscedasticity and "linearises" the data. However, from what I have > > read, transformations of the dependent variable introduces a bias into the > > regression, producing improper estimates after back-transforming to "real" > > units. Does anybody out there have any knowledge of this problem, or have > > a strategy for correcting for this type of bias? Any help would be much > > appreciated. Thanks. It depends on what you mean by a bias. The OLS regression line minimizes a certain measure of badness-of-fit over all linear fits to the data. If the dependent variable is transformed, OLS fitting is done, and the data and line are transformed back, the new fit will *not* be optimal by that criterion. On the othe rhand, it will be optimal by some different criterion. The question is, which criterion do you want and why? The answer "because all the other researchers are using it" is not adequate. If all the other researchers jumped off a cliff, etc, etc Nor is the rhetorically loaded word "bias" a reason to avoid using a method. Technically, it just means that the curve given isn't what another method would have given. The usual *informed* reason for OLS fitting is that with a homoscedastic normal error model it is the maximum-likelihood estimate for the parameters of the line. If your data do not support such an error model, then that reason doesn't apply. If after transforming the dependent variable the data *do* fit a homoscedastic normal error model, then within that family of conditional distributions the maximum likelihood choice *is* the one obtained by OLS fitting to the transformed data. In other words, the reason that often justifies OLS fitting justifies, in this case, precisely the transformed fit that you obtained. So if your transformed data fit the criteria for OLS fitting, fit them and transform back, and don't worry about "bias". -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Is there a problem with this kind of comparison?
Lucas Wells wrote: > So, what I often see, then is: > > Orders (note: presented as Aug, Sep, Oct): > > Orders Issue: 1, 9000, 9500 > Orders With Errors: 2000, 2500, 2250 > % Orders With Errors: 20%, 27.78%, 23.68% > > Fields With Errors: > > Name Field: 750, 1000, 1100 > Address Field: 750, 900, 700 > Products Ordered: 500, 550, 450 > > % Fields With Errors (ie Number of Field Errors Divided By Total > Errors for that month) > > Name Field: 37.50%, 40%, 48.89% > Address Field: 37.50%, 38%, 31.11% > Products Ordered: 25%, 22%, 20% > > Now, here's what makes me uneasy -- usually where I see a table > containing percentages of a total over a period of time, as in '% > Fields With Errors' straight above, I will also see a graph with all > three data elements plotted (ie Name Field, Address Field, Products > Ordered) across the period examined (ie Aug, Sep, Oct) with some > commentary like: > > "We can see from Sep to Oct that the percentage of errors in the Name > Field increased, however we managed to decrease the percentage of > errors in the Products Ordered field." > > Now, I look at these percentages and I think to myself, 'They're > percentages of a whole. If one goes up, then another must fall. It > doesn't seem to make sense to examine them as if they are measures > that can be seperately influenced (ie, as if we could decrease > percentages across the board).' You are absolutely correct, this is what happens when the innumerate are let loose with computers. It reminds me a bit of the thing that was circulating on the Net a while ago in which the birth year, year of taking office, years in office at the end of WWII, and age at the end of WWII were added up for each of Churchill, Mussolini, Hitler, Stalin, and FDR and - surprise! - all five sums were the same. (Diddy, diddy, diddy, diddy...) -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: One-tailed, two-tailed
[NOTE: this is CC'd to EDSTSAT-L] Stan Brown wrote: >... Now we come to the part I'm having conceptual trouble with: "Have > you proven that one gas gives better mileage than the other? If so, > which one is better?" > > Now obviously if the two are different then one is better, and if > one is better it's probably B since B had the higher sample mean. > But are we in fact justified in jumping from a two-tailed test (=/=) > to a one-tailed result (>)? > > Here we have a tiny p-value, and in fact a one-tailed test gives a > p-value of 0.0001443. But something seems a little smarmy about > first setting out to discover whether there is a difference -- just > a difference, unequal means -- then computing a two-tailed test and > deciding to announce a one-tailed result. > > Am I being over-scrupulous here? Am I not even asking the right > question? Thanks for any enlightenment. Yes, you're being over-scrupulous. The "tailedness" of the test is a matter of what p-value you are claiming; as I've argued before, under ordinary circumstances the one-tailed t test should always be avoided anyhow. The tailedness of the interpretation is a completely separate matter. It is true that there is a tiny risk of what ought to be called a "type III error" (were that not a standing but slightly misleading jocoserium referring to "testing the wrong hypothesis" or something of the kind) in which the null is wrong in one direction but the sample suggests that it is wrong in the other direction; the probability of this (conditional on the actual distribution and choice of null) is always strictly less than the p-value, so we need not be overly concerned. Moreover, if we have determined that we have enough power that we may think of the failure-to-reject outcome as implying that the difference is not practically significant (which implies outside knowledge of what constitutes practical significance), the conditional probability that we draw a sample implying ">" when the reality is "<" by a practically significant amount is tiny even compared with p. My advice, for t tests: ALWAYS test two-sided; ALWAYS interpret one-sided if you reject the null. Happy New Year! -Robert Dawson = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Looking for a proof
Moataz wrote:
>
> Dear All,
> Can anyone tell me where to find source code or even the
> algorithm for simulating The normal distribution and the Exponential
> distribution ?
Exponential distribution is easy: the negative logs of uniform random
numbers are exponentially distributed. Do trap the case X=0 before
transforming, as your RNG will probably yield this with P>0.
Normal is done using a clever trick closely related to the one used to
integrate the area under the normal curve; the idea is that if you take
the joint distribution of *two* independent normal variables in polar
coordinates, the extra "r" in the polar area element (r dr dtheta) is
precisely what you need to integrate
e^-x^2 e^-y^2 = e^-(x^2 + y^2) = e^-r^2
by substitution. The function you get after the substitution is an
exponential (which we know how to do); it follows that if U,V are
uniform on [0,1],
sqrt(-2 ln U) cos (2 pi V)
is normally distributed. Moreover,
sqrt(-2 ln U) sin (2 pi V)
is independently normally distributed; so with careful coding you get
two independent normal RV's out of this calculation for little more than
the price of one. Again, the careful programmer will trap the case
U=0.0 (note that this is one of the VERY few times when an exact test
for equality between floats is valid; alternatively you can do the
calculation within a try{} block or the equivalent) . *No* other
nonnegative value, no matter how tiny, should cause a floating-point
error on any normal system, as the ln of the least positive number on
most systems will be greater than (say) -10,000.
Finally, you ought to be sure your original uniform PRNG is good. Many
of those in standard libraries are not, having (for instance) sequential
dependence.
Moreover, if you create U[0,1] PRV's from short integer RV's (say
0...65535) you may have noticeable granularity in the far tails.
George Marsaglia has produced several excellent PRNG's, easy to code and
available in the literature.
Happy New Year
Robert Dawson
PS: To the regular posters: If EDSTAT-L were to have a FAQ list, this
question would be a good candidate.
=
Instructions for joining and leaving this list, remarks about the
problem of INAPPROPRIATE MESSAGES, and archives are available at
http://jse.stat.ncsu.edu/
=
Re: claculate L2 for Venus
Brad Guth wrote: > > My URL has much to do with the discovery of "GUTH Venus" > http://geocities.com/bradguth > > One of my questions has to deal with a manned mission, which may need > to utilize the orbit station L2, as an orbit situated so as to sustain > life onboard the spacecraft for several months to perhaps years. > > If the space craft were to be represented by upwards of 1000 tons > mass, what would the calculated L2 become? > > The same calculation at 500 tons and perhaps 100 tons. > > I've asked this of many NASA types and they exploded. Please don't > explode on my account. A good lead to an astro/gravity-calculator may > do just fine. *BANG!* Just kidding. I presume that by "the orbit station L2" you mean the (unstable) second Lagrange point of the Sol/Venus pair? The location of this point is not affected to first order by placing a mass there. The only effect would be a second-order effect due to the perturbation, due to the station's mass, of the orbit of Venus [and, if you want to get truly silly, of the position of the Sun.] As in this case "order of magnitude" is essentially the ratio of the mass of the station to that of the Sun [not of Venus, which is the object being moved...] you can see that the "empty" location of L2 would continue to be valid in the presence of any object we could put there, probably to within a micron. That said, I feel I must add something on the subject of the NASA Venus images that you think show artifacts. I do not know if you are familiar with either marquetry or gemmology. If you have any experience with the first, you will probably know about the American red gum tree (_Fluidambar_styrax_ - what a beautiful name!). Its veneer is much sought after by marquetarians, because a slice cutting through both heartwood and sapwood often contains a detailed desert scene, with cirrus clouds in the sky and sand dunes on the ground. The dunes are often even silhouetted against the skyline. There are also sedimentary rocks that are used for jewellery which, when sliced, regularly show landscapes in which the eye "recognizes" many details. Again, I have seen a poster with an entire alphabet made up of photographs of details from butterflies' wings; and while *some* resemblances between such markings and other creatures are presumably evolutionarily advantageous in that the resemblance is to something a predator will avoid, this hardly explains the sometimes near-perfect human skull seen on the Death's-Head Moth. [You will have seen it on the cover of most paperback editions of "The Silence of the Lambs". On some of these it's retouched, so that the skull itself is made up of several ghostly female bodies; but the basic image is fairly accurate.] What I'm getting at is that people tend to underestimate the ability of naturally-arising phenomena to mimic other things, without intelligent intervention. You should also be aware that these photographs were not taken using light, but by radar. If I remember correctly, they were not even taken as images, but as a linear scan pattern, assembled into an image by a computer on Earth. The bright lines are NOT differently-colored regions, or uniformly higher (or lower) than those around them; they are cliffs between terraces. If you imagine that the model is on a table top, made of dark clay, and lit from the side, it will be easier to interpret. Or think of a landscape seen at sunset. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What is the difference between Statistics and Mathematical
Jerry Dallal wrote: > > "Robert J. MacG. Dawson" wrote: > > > > Jerry Dallal wrote: > > > > > > "Robert J. MacG. Dawson" wrote: > > > > > > > > Art Kendall wrote: > > > > > > > > . Mathematical > > > > > statisticians need more course work than general statisticians. Many > > > > > agencies pay 15% more to a math statistician than a general > > > > > statistician. > > > > > > > > So what you're saying is that it isn't a difference, it's a ratio? > > > > -Robert Dawson > > > > > > > > > > I believe the current US administration is less likely to put > > > restrictions on logging. Therefore, viewed properly, it can be a > > > difference. > > > > AARRRGH!!! > > > > There are many who view the current administration's environmental > policies this way. They feel your pain! That was intended as a spotted howl. The spots didn't email well. -Robert = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What is the difference between Statistics and Mathematical
Jerry Dallal wrote: > > "Robert J. MacG. Dawson" wrote: > > > > Art Kendall wrote: > > > > . Mathematical > > > statisticians need more course work than general statisticians. Many > > > agencies pay 15% more to a math statistician than a general > > > statistician. > > > > So what you're saying is that it isn't a difference, it's a ratio? > > -Robert Dawson > > > > I believe the current US administration is less likely to put > restrictions on logging. Therefore, viewed properly, it can be a > difference. AARRRGH!!! = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What is the difference between Statistics and Mathematical
Art Kendall wrote: . Mathematical > statisticians need more course work than general statisticians. Many > agencies pay 15% more to a math statistician than a general > statistician. So what you're saying is that it isn't a difference, it's a ratio? -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Sorry for question, but how is the english word for @
Nathaniel wrote: > > Hi, > > Sorry for question, but how is the english word for @ > Pleas forgive me. You're forgiven... The New Hacker's Dictionary gives: common: at sign; at; strudel rare (and often facetious): vortex, whorl, whirlpool , cyclone, snail, ape, cat, rose, cabbage. Official ANSI name: commercial at -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: When to Use t and When to Use z Revisited
Ronny Richardson wrote: > > Are they > > 1. Wrong > 2. Just oversimplifying it without telling the reader Neither, really. The MAIN objection to "z over 30" is that it adds an an unnecessary step to the decision process. If it actually simplified things greatly I reckon we could live with the slightly wonky p-values (as we do when we use ANOVA in the knowledge that we do not have perfect homoscedasticity). But it makes things more complicated... A true cynic might say that there is one advantage to keeping the procedure in the textbooks - it wil occasionally give the readers of articles warning that the writer has learned statistics by rote. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Gen random numbers from distribution
Jim Snow wrote: > > 1. George Marsaglia and Wal Wan Tsang published a paper dealing with > your problem which gives an efficient procedure for all values of > parameters. It is > > "The Monty Python Method for Generating Gamma Variables" > > in the Journal of Statistical Software ,vol3,issue 3,1998 > . > > This is an online journal. The paper is available at > > www.jstatsoft.org/v03/i03/ ^^^ you'll need this! Thanks for the tip, it's a good paper. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Interpreting p-value = .99
Stan Brown wrote:
I see why the quality controller would want to
> do a two-tailed test: the product should not be outside
> manufacturing parameters in either direction. (Presumably the QC
> person would be testing the pills themselves, not patients taking
> the pills.)
Actually, the quality controller's test is a slight misnomer here,
because we aren't talking in this problem (as you more or less observed)
about standard QC methodology. Standard QC doctrine, from what I hear,
generally goes for repeatability, and "better than specified" is *not*
good. ("So, how did you do in the QC Methods exam?" "My score was three
sigmas above the class average... so the prof failed me.")
The question dealt with a situation, though, in which only one
direction of deviation is bad. Thus, the test might legitimately be
one-sided. The reason is that the alpha value represents the risk of
unnecessarily stopping the production line, reprinting the labels, or
whatever. You *don't* need to do this if the product works better than
advertised, so a one-sided alpha really is the risk of doing it
unnecessarily.
> But I don't see why either the advertiser or the consumer advocate
> would, or should, do a two-tailed test.
The idea is that the "product" of these tests is a p-value to be used
in support of an argument. The evidence for the proposal is not made any
stronger by the tester's wish for a certain outcome; so the tester
should not artificially halve the reported p-value.
Superficially, the idea of halving your p-values, doubling your chance
of reporting a "statistically significant" result in your favored
direction if there is really nothing there, and as a bonus, doing a
David-and-Uriah job ("And he wrote in the letter, saying, Set ye Uriah
in the forefront of the hottest battle, and retire ye from him, that he
may be smitten, and die.") on any possible finding in the other
direction, may seem attractive. A moment's thought should persuade us
that it is not ethical.
-Robert Dawson
=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=
Re: Interpreting p-value = .99
Stan Brown wrote: > > On a quiz, I set the following problem to my statistics class: > > "The manufacturer of a patent medicine claims that it is 90% > effective(*) in relieving an allergy for a period of 8 hours. In a > sample of 200 people who had the allergy, the medicine provided > relief for 170 people. Determine whether the manufacturer's claim > was legitimate, to the 0.01 significance level." > A hypothesis test is set up ahead of time so that it can only give a definite answer of one sort. In this case, we have (at least) three distinct possibilities. (0) The "advertiser's test": we want a definite result that makes the manufacturer look good, so Ha agrees with the manufacturers claim. To do this the manufacturer's claim must be slightly restated as "our medicine is *more* than 90% effective"; as the exact 90% value has prior probability 0 this is not a problem. H0 is actually the original claim; and the hoped-for outcome is to reject it because the number of successes is too large. The manufacturer is not entitled to do a 1-tailed test just to shrink the reported p-value. Using a 1-tailed test is to say "I want all my Type I errors to be ones that let us get away with inflated claims." This is what the students did: > But -- and in retrospect I should have seen it coming -- some > students framed the hypotheses so that the alternative hypothesis > was "the drug is effective as claimed." They had > Ho: p <= .9; Ha: p > .9; p-value = .9908. Ethical behaviour is to do a two-tailed test, and report/act on a rejection in either direction. It is not necessary to say "there is a difference but we don't know in which direction"; a two-tailed test can legitimately have three outcomes (reject low, reject high, not enough data). (There is a potential new type of error in which we reject in the wrong tail; this *ought* to be called a Type III error were the name not already taken [as a somewhat misleading in-joke akin to the "Eleventh Commandment"] to mean "testing the wrong hypothesis" or something similar. It is easy to show that the probability of this is low enough to ignore if alpha is even moderately low, as the distance between tails is twice the distance from the mean to the tail.) (1) The "consumer advocate's test": we want a definite result that makes the manufacturer look bad, so H0 is the manufacturer's claim, Ha is that the claim is wrong, and the p-value is to be used as an indication of reason to believe H0 wrong (if so). Using a one-sided test here is akin to saying "I want all my type I errors to be ones that make the manufacturer look bad". Ethical behaviour here is to do a two-sided test and report a result in either direction. (2) the "quality controller's test": H0 is the manufacturer's claim, Ha is that the claim is wrong, and the p-value is to be used to balance risks. Here, I think, a one-tailed test is legitimate. I claim that the consumer advocate and the manufacturer *should* be doing the same test in situations 0 and 1. Both should be reporting a p-value of 0.0184, both should be interpreting it as "the medicine is less effective than claimed", and the manufacturer should take action by either improving the product or modifying the advertisements. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Normal distribution
Ludovic Duponchel wrote: > > If x values have a normal distribution, is there a normal distribution > for x^2 ? No. If the mean is 0, x^2 hasa chi-squared distribution with 1 DOF. As the ratio mean/SD -> infinity, the distribution of x^2 is asymptotically normal. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: N.Y. Times: Statistics, a Tool for Life, Is Getting Short Shrift
Speaking of normal distributions and cancer clusters, does anybody (a) agree with me that the human race in general has a better "feel" for the normal distribution than the binomial distribution, and the Poisson is still worse - and (b) know of any experimental evidence for this? That is, my conjecture is that if an untrained human thinks that there is an unusually large collection of tall people, or larger-than-usual apples, or whatever, in a collection, they are probably right; but there is a tendancy to expect more uniformity in Bernoulli and Poisson processes than should be there. People tend to see clusters of things and streaks of events when they are not really there. There is probably a reverse trend in the extreme tail; people probably overestimate the probability of getting (say) red fifty times in a row at Roulette simply because we don't have a good feel for really large and small numbers. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: t vs. z - recapitulation
Gaj Vidmar wrote: > > sample size | distribution(s) | population var | appropriate test > > -- > large (say, N>30) | normal | known | z (obvious) No, here "large" is irrelevant. N=1 can be used... > large | not normal | known | z (CLT takes care of > numerator) N>30 may not be enough; or N=10 may be fine. > small | not normal | known | still z, right?? You may need a made-to-order test here, and *will* if you define "small" as "too small for the CLT to help" > large | normal | estimated | t (note 1 below) > small | normal | estimated | t (the case of > Student) > small | not normal | estimated | mostly t (note 2 > below) > > Note 1: z before computer era and also OK due to Slutsky's theorem Not necessary as a separate test even then, as interpolating between the n=100 and n=infinity rows of the t table is intuitive and avoids an unnecessary choice. Traditional t tables did not have enough alpha levels but that's another story (see http://www.amstat.org/publications/jse/v5n2/dawson.html for an alternative; and some books now use a similar table (eg, Devore & Peck). > > Note 2: t-test is very robust (BTW, is Boneau, 1960, Psychological Bulletin > vol. 57, referenced and summarised in Quinn and McNemar, Psychological > Statistics, 4th ed. 1969, with the nice introduction "Boneau, with the > indispesable help of an electronic computer, ...", still an adequate > reference?), whereby: > - skewness, even extreme, is not a big problem > - two-tailed testing increases robusteness > - unequal variances are a serious problem with unequal N's with larger > variance of smaller sample > > Now, what to do if t is inadequate? - This is a whole complex issue in > itself, so just a few thoughts: > - in case of extreme skewness, Mann-Whitney is not a good alternative > (assumes symmetric distrib.), right? For two samples, Wilcoxon-Mann-Whitney assumes symmetry *or* shift alternative *or* transformability to one of these (eg, lognormal). For paired and one-sample data, there's always the sign test; it assumes practically nothing. Actually, differences of paired data are a pretty good bet for symmetry if the populations are at all similar in shape. (If they are not, one should think long and hard about whether the comparison means anything anyway.) Thus one can usually use signed-ranks on the differences. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: N.Y. Times: Statistics, a Tool for Life, Is Getting Short Shrift
The NY Times wrote: > It is no longer possible to serve competently on some juries >without more data skills than most college graduates have. That's all right, there will always be one lawyer or the other who doesn't *want* anybody to serve competently, and the competent juror will be challenged. Having an above-average knowledge of anything relevant (especially the law) has always been an easy route to disqualification from jury duty. At least, I'm assuming that that's why lawyers are automatically disqualified in many jurisdictions. Only a cynic would assume that it's just that the legal system has decided that if you make that kind of money you're too important to have to take a week's unpaid leave; and only an idealist would assume that members of a profession, 50% of whom are at any moment trying to lead a jury to either acquit a guilty person or convict an innocent one, would on that account not be let near the jury box! > If the trend continues nationwide, this newspaper could someday report > that an apparently alarming cluster of cancer cases has arisen in an > innocuous normal distribution, and students will be able to explain to > their parents what that means. Well, I'm darned if I can... "clusters" of cancer cases generally arise either as spatial, temporal, or categorical data, and a normal distribution would be unlikely to apply. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Need help with a probability problem
> On 20 Nov 2001, J. Peter Leeds wrote: > > The problem actually breaks down to a rather simple analogy: > > > > Imagine that a man has been sentenced by court to run a gauntlet > > composed of four club-wielding executioners. (ill-defined, and thus insoluble, problem omitted) and Donald Burrill responded: > Easier it may be, but one can't help suspecting that some aspects of the > inanities evident are not paralleled by structures or relationships in > whatever your "real" problem is... I agree with Donald...reading EDSTAT-L, I am continually reminded of the cliche about the patient at the doctor's clinic who "has a friend who thinks (s)he might have VD". It seems as if many correspondents posting "real" (non-homework) problems go to great lengths to "anonymize" their problems, usually to the extent of making sensible advice impossible. Some possible explanations: 10) They don't want their colleagues to know they've consulted EDSTAT-L. 9) We might steal their data. 8) Statisticians are like mushrooms - they do best when kept in the dark and fed well-composted BS. 7) EDSTAT-L is the favorite consulting service for researchers on witness protection programs. 6) There is actually no research project as described; rather, this is part of an elaborate psychological study of mailing list subscribers. 5) They know that editors don't like putting the entire list of EDSTAT-L posters as coauthors. 4) They reckon we wouldn't understand the difficult stuff they're working on & don't have time to explain it. 3) Some researchers realize that they are working on subjects of truly embarrassing stupidity, but they need the grant money. 2) They want to do as much of the work as they can for us and think that this is the first step. And the top reason: 1) These posters are from the NSA: they could tell us but then they would have to kill us. Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: diff in proportions
> Jerry Dallal wrote: > >But, if the null hypothesis is that the means are the same, why >isn't(aren't) the sample variance(s) calculated about a pooled >estimate of the common mean? I looked at this some years ago. The answer is straightforward: it would be logically valid to do so but you would lose a *lot* of power. A hypothesis test is essentially a proof by contradiction; in such an argument you are permitted to run with the hare and hunt with the hounds, changing sides as often as you like. Thus, at any stage, you may appeal to the null hypothesis or to the data; any inconsistency between the two, no matter how byzantine the argument, is evidence against the null. If you think about the two-sample-T as a two-level ANOVA (a roughly correct idea), the pooled estimate of the mean gives you the SST; the usual method gives you the SSE. As you expect the SSTr to be nonzero, you have SSE < SST and substituting one for the other is a Bad Thing. In an extreme case: A B 10 20 11 21 12 22 one method estimates the SD as 1, the other as 5.55. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: diff in proportions
Dennis Roberts wrote: > > At 08:51 AM 11/15/01 -0600, jim clark wrote: > > >The Ho in the case of means is NOT about the variances, so the > >analogy breaks down. That is, we are not hypothesizing > >Ho: sig1^2 = sig2^2, but rather Ho: mu1 = mu2. So there is no > >direct link between Ho and the SE, unlike the proportions > >example. > > would it be correct then to say ... that the test of differences in > proportions is REALLY a test about the differences between two population > variances? No, because it would reject the null (with large enough samples) when pi_1 = 1-pi_2, despite the fact that the variances would be equal! -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: definition of " metric" as a noun
"No Spam Mapson" wrote: The OED cites the following use of metric as a noun: 1921 Proc. R. Soc. A. XCIX. 104 "In the non-Euclidean geometry of Riemann, the metric is defined by certain quantities ... >>> >>> A good example of bad usage: *what* metric, *what* quantities? >>> The reader should not be left hanging with those questions unanswered. >> >> This is not bad usage at all. In mathematics, the word >> "metric" as a noun refers to a general type of distance, >> not necessarily the type in common use. > > It is certainly bad usage, for the following reason: the phrase, > "the metric", implies that there is *one* metric function on > Riemannian geometry, which is false. This reason has nothing > to do with distance measure in general, as commonly understood, > or otherwise. It is not bad usage, because a PARTICULAR Riemannian geometry is given by a particular metric; in fact, by the local quadratic form defining the differential metric. >>> >>> It *is* bad usage, because it requires the type of exegesis you >>> have just provided to make it meaningful. >> >>Statisticians, of course, always use completely unambiguous terminology > > This is not about terminology; this is about usage. > No, this is about setting the world record for the longest string of ">"s. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Can I Use Wilcoxon Rank Sum Test for Correlated & Clustered Data??
Chia C Chong wrote: > > I am a beginner in the statistical analysis and hypothesis. I have 2 > variables (A and B) from an experiment that was observed for a certain > period time. I need to form a statistical model that will model these two > variables. As an initial step, I plot the histograms of A & B separately to > see how the data were distributed. However, it seems that both A & B can't > be easily described by a simple statistical distributions like Gaussian, > uniform etc via visualisation. Hence, I proceeded to plot the > Quantile-Quantile plot (Q-Q plot) and trying to the fit both A and B with > some theoretical distributions (all distributions avaiable in Matlab!!). > Again, none of the distributions seem can descibe then completely. Then I > was trying to perform the Wilcoxon Rank Sum test. From the data, it seems > that A & B might be correlated in some sense. If the data are (positively) correlated, do not use the Wilcoxon-Mann-Whitney rank sum test; use the sign test on the differences, which will usually be much more powerful in the presence of significant correlation. If the two populations differ (roughly) only by translation, the differences may well be (roughly) symmetrically distributed. Then you may get more power yet by using the signed ranks test on the differences (confusingly, this is also named for Wilcoxon). IN MINITAB: (data in C1, C2) let C3 = C1-C2 wtest c3 = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Are parametric assumptions importat ?
Glenn Barnett wrote:
> > (1) normality is rarely important, provided the sample sizes are
> > largish. The larger the less important.
>
> The a.r.e won't change with larger samples, so I disagree here.
I don't follow. Asymptotic relative efficiency is a limit as sample
sizes go to infinity; so how does it change or not change "with sample
size"? Or does that acronym have another expansion that I can't think
of?
I hadn't had efficiency in mind so much as the validity of p-values for
the t test. However, the same point holds for efficiency. For large
samples, I would suggest that the efficiency of both tests is usually
adequate; and a small sample does not tell us enough about the
population distribution to tell much about the relative efficiency
anyway.
When you've got lots of data, you also have a choice of lots of
reliable methods of inference; when you haven't got enough, you also
can't trust the methods that look as though they might help. ("Sort of a
metaphor for life", he said cynically.)
You are of course right that when I wrote "rank-sum" I meant
"signed-rank". (Of course, as Potthoff showed, the rank-sum test *is*
valid under the assumption of symmetry, but that is another story.)
-Robert Dawson
=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=
Re: Help for DL students in doing assignments
Nomen Nescio wrote:
>
> >Mr. Dawson wrote:
>
> >Well, they do say what goes around comes around; I'd love to see what
> >mark the dishonest DL student gets having had his homework done for him
> >by somebody who:
> >
> >(a) believes all primes to be odd;
> >...
> ### Let's assume that any prime is NOT odd
> ### It means that is is even (no other way among integers!)
> ### So that prime has 3 dividers: "1",this prime and "2"
> ### which contradicts with prime definition:
> ### ("prime is integer that has only two dividers: 1 and this prime itself")
> ### Dear Mr. Dawson, please send me at least ONE even prime
> ### and i shall give you $1,000,000.
2.
Da mihi pecuniam.
-RD
=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=
Re: Help for DL students in doing assignments
Well, they do say what goes around comes around; I'd love to see what mark the dishonest DL student gets having had his homework done for him by somebody who: (a) believes all primes to be odd; (b) believes that A=>B implies B=>A; (c) has never heard of Goldbach's conjecture; (d) expects despite this to pass as a PhD in mathematics among an audience containing many such. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Final Exam story
Donald Burrill wrote: > > "The story is about six students who ... The instructor ... tells them > to report the next day for an exam with only one question. If they all > get it right they all pass. They were seated at corners of the room and > could not communicate." > > Must have been an interesting room, with six corners :) > > "The one question was, "Which tire?" I remember that the likelihood of > all four pickng the same tire was quite small, but I forgot how to > calculate it explicitly." > > Assuming an ordinary vehicle with 4 tires, and that the students' > responses are independent, (1/4)^6 = 1/4096. No, (1/4)^5. [See my first posting in which I mistakenly solved for four students... but the principle that one student's guess is "free" still holds.] -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Final Exam story
Dubinse wrote: > > I had promised a colleague a story that illustrates probability and > now I forgot how to solve it formally. The story is about six > students who go off on a trip and get drunk the weekend before > their statistics final. They return a few days late and beg for a > second chance to take the final exam. They tell a story about how > they were caught in a storm and their car blew a tire and ended up > in a ditch and they needed brief hospitalization etc. The instructor > seems very easy going about the whole thing and tells them to report > the next day for an exam with only one question. If they all get it right > they all pass. They were seated at corners of the room and could not > communicate. The one question was, "Which tire?" I remember that > the liklihood of all four pickng the same tire was quite small, but I > forgot how to calculate it explicitly (except for listing all the possible > outcomes). Pick one student (call her Alice). By symmetry it doesn't matter which tire she picks. Then they pass if Betty, Charlene, and Dianne each independently pick that tire, which each has 1 chance in 4 of doing; so the answer is 1 in 4x4x4 = 64. (On the other hand if they are in political science or international relations and have read Schelling on "The Strategy of Conflict", they will probably all pick the left front tire (or right front in Britain), as the most "distinct" one. In fact, experiments have shown that this sort of spontaneous agreement *does* occur even without special training!) -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Bimodal distribution
Desmond Cheung wrote: > > Is there any mathematical analysis to find how much the two peaks stand > out from the other data? Is there any formulas to find the > variance/deviation/etc that's similar to the unimodal distribution case? In answer to the latter question - excatly the ones you use in the unimodal case. If you want to find the breadth of *one* peak, that is more difficult and in general there is no canonical way to do so. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Are parametric assumptions importat ?
Voltolini wrote: > > Hi, I am Biologist preparing a class on experiments in ecology including > a short and simple text about how to use and to choose the most commom > statistical tests (chi-square, t tests, ANOVA, correlation and regression). > > I am planning to include the idea that testing the assumptions for > parametric tests (normality and homocedasticity) is very important > to decide between a parametric (e.g., ANOVA) or the non parametric > test (e. g. Kruskal-Wallis). I am using the Shapiro-Wilk and the Levene > test for the assumption testing but.. It's not that simple. Some points: (1) normality is rarely important, provided the sample sizes are largish. The larger the less important. (2) The Shapiro-Wilk test is far too sensitive with large samples and not sensitive enough for small samples. This is not the fault of Shapiro and Wilk, it's a flaw in the idea of testing for normality. The question that such a test answers is "is there enough evidence to conclude that population is even slightly non-normal?" whereas what we *ought* to be asking is "do we have reason to believe that the population is approximately normal?" Levene's test has the same problem, as fairly severe heteroscedasticity can be worked around with a conservative assumption of degrees of freedom - which is essentially costless if the samples are large. In each case, the criterion of "detectability at p=0.05" simply does not coincide withthe criterion "far enough off assumption to matter" except sometimes by chance. (3) Approximate symmetry is usually important to the *relevance* of mean-based testing, no matter how big the sample size is. Unless the sum of the data (or of population elements) is of primary importance, or unless the distribution is symmetirc (so that almost all measures of location coincide) you should not assume that the mean is a good measure of location. The median need not be either! (4) Most nonparametric tests make assumptions too. The rank-sum test assumes symmetry; the Wilcoxon-Mann-Whitney and Kruskal-Wallis tersts are usually taken to assume a pure shift alternative (which is actually rather unlikely for an asymmetric distribution.) In fact symmetry will do instead; Potthoff has shown that the WMW is a test for the median if distributions are symmetric. If there exists a transformation that renders the populations equally-distributed or symmetric (eg, lognormal family) they will work, too. In the absence of some such assumption strange things can happen. I have shown (preprint available on request) that the WMW test is intransitive for "most" Behrens-Fisher families (that is, it can consistently indicate X>Y>Z>X with p -> 1 as n -> infinity), although the intransitivity is not pronounced for most realistic distributions and sample sizes. Note - a Behrens-Fisher family is one differing both by location and by spread but not by shape. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: ranging opines about the range
> > One what? Any statistic that depends on the units used seems rather > >arbitrary to me. If I compute the range of weights of a group of people > >(in kilograms) I ought to get the same actual *weight* as an American > >using pounds or a Brit using stones. > > On a lighter note - sorry - Brits can't use stones as however > reluctantly we are now metricated. Selling things in pounds and stones > is against the law - though I suppose that using the measure is not - > yet! I just tell students I'm B.C. - Before Centimetres. The main thing measured in stones surely used to be people (as in my example) and selling people has been highly illegal for a long time now. -Robert = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: ranging opines about the range
"William B. Ware" wrote: > > Robert, > > I don't think I understand your argument... Are you saying that the > "descriptive statistic" should be invariant over scale? Yes, of course. Otherwise it's a description of the process of measuring, not of what is being measured. > > Anyway, more to the point... the "add one" is an old argument based on the > notion of "real limits." Suppose the range of scores is 50 to 89. It was > argued that 50 really goes down to 49.5 and 89 really goes up to > 89.5. Thus the range was defined as 89.5 - 49.5... thus the additional > one unit... Yes, but not all data are integer test scores. *If* you are given data that have been rounded or truncated to the nearest multiple of some D (in your example, 1), then I suppose that an unbiased estimator of the original range would be max + 1 - min or something like that. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: ranging opines about the range
jeff rasmussen wrote: > > Dear statistically-enamored, > > There was a question in my undergrad class concerning how to define the > range, where a student pointed out that contrary to my edict, the range was > "the difference between the maximum & minimum". I'd always believed that > the correct answer was the "difference between the maximum & minimum plus > one" One what? Any statistic that depends on the units used seems rather arbitrary to me. If I compute the range of weights of a group of people (in kilograms) I ought to get the same actual *weight* as an American using pounds or a Brit using stones. Suppose I have three meter sticks - are you telling us that the range of their lengths is a little over one meter? I'm afraid I vote with your students. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Help
david007 wrote: Let A denote the maximum run length, i.e. the > largest number of consecutive heads we get among the n tosses. > > Find P(A=2) analytically (not by simulation) for the case n=5. Try rephrasing. "A=2" means that (1) there are two heads in a row and (2) they are preceded and followed either by the end of the trial or by tails. Now count the ways for this to happen (checking for double counting; this is vacuous for n=4 and trivial for n=5, but gets less trivial for larger n.) For a really large n, I think I'd try a different tack, and either subtract P(A<=1) from P(A<=2) or subtract P(A>=3) from P(A>=2); these would need to be calculated by recursion. For instance, to find P(A <= 1) for n, that is the probability of no pairs of heads, you'd split it into the two disjoint events E_n:no HH in first n, X_n = H [hence X_(n-1=T)] e_n:no HH in first n, X_n = T Now, E_(n+1) = 0.5 e_n e_(n+1) = 0.5 e_n + 0.5 E_n and E_1 = e_1 = 0.5 and we get E_n = F_n/2^n, e_n = F_(n+1)/2^n (proof omitted) where F_i is the ith Fibonacci number. and, summing, P(A <= 1) = F_(n+2)/2^n . A similar calculation would give us P(A<=3). This would be cracking a nut with a sledgehammer for n=5, though. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Help for DL students in doing assignments
Nimish Shah wrote: > > > Dear DL Students, > > > > I have Ph.D. degree in mathematics, physics, electrical engineering, > > computer science. > > Can you please post details of your 3 PhDs! Four, surely? -R. Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: CIs
Dennis Roberts wrote: > > it seems to me that the notion of a confidence interval is a general > concept ... having to do with estimating some unknown quantity in which > errors are known to occur or be present in that estimation process > > in general, the generic version of a CI is: > > statistic/estimator +/- (multiplier) * error Dennis, this is just not so. Even some familiar and widely-used confidence intervals like the sign interval do not follow this pattern. I think I understand what you are trying to say; there is a general concept of interval estimator, and within that a special class which are obtained by specifying a point estimator for a parameter and some multiple of its spread. *Some* confidence intervals - the most familiar, I will agree - are of this type. But not all of these are confidence intervals and not all CI's are of this type. So please let us not abuse the word "confidence interval" for things that generally aren't. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: definition of " metric" as a noun
"Robert J. MacG. Dawson" wrote: > Actually, there *is* essentially one canonical metric function on > Riemannian geometry. In either model of absolute geometry there is, up > to a multiplicative constant, only one metric preserved by reflection. > In hyperbolic geometry, moreover, there is an absolute distance scale > based on angular defect. I should, of course, have said "one metric which is additive along lines". The discrete metric (for instance) is preserved by reflection. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: definition of " metric" as a noun
Emord Nila Palindrome wrote: > It is certainly bad usage, for the following reason: the phrase, > "the metric", implies that there is *one* metric function on > Riemannian geometry, which is false. This reason has nothing > to do with distance measure in general, as commonly understood, > or otherwise. Actually, there *is* essentially one canonical metric function on Riemannian geometry. In either model of absolute geometry there is, up to a multiplicative constant, only one metric preserved by reflection. In hyperbolic geometry, moreover, there is an absolute distance scale based on angular defect. Obviously, you can put a different metric on the points of the hyperbolic plane (or on any set); but this is a vacuous observation. Only one metric is actually a metric for the plane rather than its underlying set. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: what type of distribution on this sampling
Rich Ulrich wrote: > > > > But what if I have a population of numbers that is made up of > 1 billion draws from a Cauchy distribution? No one has ever > defined this for me, and I have never tested it, but if you *have* > a sample in hand, then you *can* compute a standard deviation > even though in theory the variance is "undefined" or infinite. > > I does seem to me that you could take your billion numbers, and > standardize them to the mean of 78 and SD of and > it won't change the nature, that the numbers belong to a Cauchy - > and repeated sampling will *not* give a standard deviation-of- > the-mean that is smaller than the standard deviation of the draw. > > I'm afraid that I am not giving an answer here - I am raising a > question that I don't know the answer to. > > But -- without being Cauchy -- it must be true that you can always > imagine an outlier extreme enough that your means will not > be normal, or be very useful compared to some other description. *Numbers* do not belong to any distribution whatsoever. Numbers in the context of a method of generation do. If you standardize your billion numbers to a specified mean and standard deviation, and then apply the same scaling to further samples the standard deviations of your other samples will not resemble that of your first sample. Resampling from the original set *will* give SD's that converge to that of the original if the derived samples are large enough. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: what type of distribution on this sampling
> Joe Galenko wrote: > > > Just out of curiousity, I'd like to know what kind of population you could > > have such that a sample mean with N = 200 wouldn't be approximately > > Normally distributed. That would have to be a very, very strange > > distribution indeed. and Gus Gassmann responded: > You can construct them easily as Bernoulli distributions where p is sufficiently > close to 0 or 1. A more concrete example: lottery ticket payouts. A batch of 200 lottery tickets still has a payout distribution heavily influenced by small proportions of very unlikely events (jackpots). -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Free program to generate random samples
> >"@Home" wrote: > >> > > >> > Is there any downloadable freeware that can generate let's say 2000 random > >> > samples of size n=100 from a population of 100 numbers. > >> > > >> > >and Randy Poe responded: > >> Um. > >> > >> A sample of 100 from a population of 100 is going to > >> give you the entire population. I replied: > > Depends whether you sample with or without replacement. and Jon Cryer wrote: > > But it would be bad statistics to sample with replacement. If I were taking one sample of size 50 from a population of voters it would probably be bad statistics to sample with replacement, because the sample would be less informative than one I could easily take without replacement. In other circumstances a sample with replacement might be the only procedure easily available; this can certainly be handled by slight (often negligible) modifications ofthe theory. But if somebody is taking 2000 samples from a population of "100 numbers" we can be fairly sure that this is for some sort of simulation purpose; many resampling techniques *require* a sample with replacement. It cannot, then, be called "bad statistics" in general. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: how to compare generated values with the specifieddistributionbasis
Rich Ulrich wrote:
> Robert waffles by saying 'most' purposes, so I have to
> find it easy to agree. When might you *not* treat a uniform,
> N=20 as normal? - perhaps when the R^2 is too high
> (above .90)?
Anything involving extreme-value estimation, for a start.
|| || ||
|| || ||
|| || ||
|| || || indeed.
-Robert Dawson
=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=
Re: Free program to generate random samples
"@Home" wrote: > > > > Is there any downloadable freeware that can generate let's say 2000 random > > samples of size n=100 from a population of 100 numbers. > > > and Randy Poe responded: > Um. > > A sample of 100 from a population of 100 is going to > give you the entire population. Depends whether you sample with or without replacement. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: for students (biology et al.) that hate numbers
Jerry Dallal wrote: > > > You can have them count the colors of candies in bags of M&Ms. The M&M > > web site has the expected proportions published so they can do a > > ChiSquare test against those proportions. > > Does anybody really care about the proportions of different colors > in bags of M&Ms? No, but they like to eat the subjects. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: how to compare generated values with the specifieddistributionbasis
Jon Cryer wrote: > > Robert: > > "even when N=20, a uniform distribution can be treated as > >normal for most purposes." > > I assume you meant to say that for N=20, the sample mean based on a random > sample from a uniform distribution can be assumed to have a normal > distribution > for most purposes. > > Right? Yes; I was writing in the context of standard inferential techniques. (t tests, etc) -Robert = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: how to compare generated values with the specified distributionbasis
JHWB wrote: > > Hm, hope I didn't make that subject to complex, resulting in zero replies. > But hopefully you can answer this: > > I have a N(20,5) distribution and based on that I generated 25 values using > Minitab and the Calc>Random data>Normal function. The result yielded a mean > of 19,083 and a standard deviation of 6,0148. > > Now, how can I compare these results numerically and graphically? > > I mean, in the back of my head I have an image of a graph with a straight > line (the basis for the values) and the plotted dots of the actual generated > data following the line. What you have in mind is a normal scores plot. This appears under Graph > Probability Plot as the default option. The hypothesis tests that come with this should be treated with great caution (though testing the output of a RNG supposed to yield normally distributed output *is* perhaps a valid use!) People have been known to do one of these tests to see whether they must use a nonparametric inference technique such as a sign or WMW test on a certain data set. However, the hypothesis test answers the question "are these data improbable under conditions of perfect normality?" and by extension "is there evidence against perfect normality?". What you want to ask is "are these data probably from a population that is normal enough for the method to work?" The gotcha is that while these may be roughly equivalent questions for (say) N=20, for N small deviations from normality are important and the test is poor at detecting them; for N large, deviations from normality do not matter very much but the test is hypersensitive. For instance: even when N=20, a uniform distribution can be treated as normal for most purposes. However, it will generally fail the Ryan-Joiner test at a 5% level! -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Tera-fying names
> The implausibly named a†a†a†a†a†a†a†a†a† wrote: > > > > I would like to ask how to convert teragram to kilogram. > > Thanks for helping~ > > Try http://www.tuxedo.org/~esr/jargon/jargon.html#quantifiers for a good rundown on this. Or multiply by 10^9. ( By the way, a†a† (may I call you that for short?): your handle seems too long for the edstat mailer so I can't reply directly. Hope you're following the thread...) -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: teragram
Sorry, I misread the original and sent the teragram -> gram conversion. D'oh! Teragram -> kilogram is of course 10^9. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: how to compare these 2 functions(asymptotically)
kmswys wrote: > > lg*(lg(n)) and lg(lg*n) > > lg* is iterated logarithm (base 2), defined as the smallest i such that > ith iteration of logarithm is less or equal one. lg*(lg(n)) is just lg*(n) - 1, asymptotically ~ lg*(n) The second expression is thus asymptotically the log of the first. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Assistance
Jay Larman wrote: > > This is a SCAM. Do not fall for it. On the contrary. If anybody reading this is (which I doubt) the sort of person who would get involved in laundering money stolen by a deceased politician in a poor country if the opportunity really arose, then I urge them: please *do* fall for it. Send the recently-discovered natural son of Field Marshal President Idi Amin Dada all your savings and the keys to your car. My only regret would be that the *other* swindler would unavoidably be enriched by the deal. But that's life. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Definitions of Likert scale, Likert item, etc.
Alan McLean wrote: > The composite variable or measure (hopefully) has a reasonably > numeric scale. I don't think (in light of the Central Limit Theorem) that the problem is whether the composite's "reasonably numeric". It is. The problem, when the data's given the usual ANOVic treatment, is what the heck the mean means. In some circumstances the only important thing is how many people "strongly agree" and the Laodiceans can be lumped with the naysayers. In other circumstances other interpretations apply. Taking the arithmetic mean of "Likert" data fudges this issue. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Simple Probability Proof Requested
> Sloppy Joe <[EMAIL PROTECTED]> wrote: > >Greetings - > > >Suppose I have a method such as a fair 6-sided die. I roll the die 10 > >times and get the following trial history: > > >3-5-1-3-6-4-6-2-1-5 > > >From what I can recall from probability, I cannot predict the next roll > >of the die based on the previous rolls. Each number on the die will > >have an equal probability of occuring on the 11th roll no matter what > >has occurred previously: 1/6. My question is this ... > > >How can this be proven mathematically? It can't be. It's just the conventional definition of "fair". There is no reason why a die cannot be weighted so that the probability of a certain number appearing on any roll is *not* 1/6; and there is no reason why a die cannot be constructed (say, using some cunning clockwork, or lead shot in a very viscous liquid) so that the probabilities for one roll depend on the rolls before it. Neither of these is ruled out by the observation described above (though they provide some evidence against the "repeating" die last described). Moreover, a die may have equal probability without independence, or independence without equal probability. "Fairness" is generally understood as implying both properties. (This may not quite correspond to the general meaning. If I say "I have a weighted die in my pocket that only rolls one number. I bet you 5 to 1 that you cannot guess what it will roll!" that is a fair bet - once! Moreover, it is a fair bet even if I do not tell you that the die is loaded, provided its faces only have conventional numbers of spots (1 through 6, though not necessarily one of each.)) -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Bimodal distributions
Dennis Roberts wrote: > major mode and the other as the minor mode. > > this is an interesting point but, one we have to be careful about ... in > the minitab pulse data set ... c6 is heights of 92 college students ... IIRC, the difference between male and female mean height is almost exactly two standard deviations. Now, it's an easy exercise in first-year calculus to show that a 50-50 mix of N(mu1,sigma^2) and N(mu2,sigma^2) has two distinct humps iff |mu1-mu2| > 2 sigma. So in fact the human height distribution ought to be just about at the critical point where there is one very flat-topped hump on the point of fission. There is a sketch in Freedman. Pisani, Purve, and Adhikari (ex. 7, p. 32, 2nd edn) in which it is suggested that the dip should be highly visible... but as Dennis has pointed out this is a fallacy. People tend to expect the mixture of two distributions to look like the superposition! Another fascinating example is the dinosaur height histogram in the book "Jurassic Park" in which the superposition is clearly used for what is supposed to be the mixture. The graph presented is not only multimodal but has nodes between the modes! In fact, that page (especially in the first edition - some of the grosser errors were corrected in the post-movie edition) has about half-a-dozen separate errors. The oddest is that the height of the peaks *increases* linearly with the height (& age) represented - somehow each clutch of dinosaurs mysteriously recruits more agemates as time goes on, rather than having their numbers reduced by accidents, predation, etc. -Robert Dawson (PS: Crichton did mention the use of frog DNA as being behind some problems in the Park; and there *is* a beast called the "paradoxical frog", mentioned in one of Gerald Durrell's books, that goes from an enormous tadpole to a very much smaller frog. But I doubt that that was intended.) = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Bimodal distributions
Dennis Roberts wrote: > > At 02:04 PM 8/30/01 -0400, David C. Howell wrote: > >Karl Wuensch asks an interesting question, though I would phrase it > >somewhat more generally. "At what point does a bimodal distribution become > >just a distribution with two peaks?" I thought about this one once (possibly "out loud" on edstat-l?). It seems to me that the useful distinction is when a minor peak becomes small enough to plausibly not exist in the underlying population/distribution. To determine this without any parametric models, starting with a histogram, one would presumably find the best fit among all unimodal *functions* [constrained to be monotone increasing to some point, thereafter monotone decreasing] using a least-chisquare criterion, then use a chisquare goodness-of-fit test with some appropriate number of degrees of freedom. There should also be a continuous version of the test which works on ungrouped data and does not throw away information by binning. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Bimodal distributions
"Paul R. Swank" wrote: >...In the bimodal case, some refer to the higher "hump" as the > major mode and the other as the minor mode. Followed by Dorian, Ionian, Lydian, Hypodorian, Myxolydian... etc? -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Boston Globe: MCAS results show weakness in teens' grasp of
I wrote: > Er, no. > > Q1 ~ mu - 2/3 sigma > Q3 ~ mu + 2/3 sigma > 1 IQR ~ 4/3 sigma > 1.5 IQR ~ 2 sigma > > inner fence ~ mu +- 2 2/3 sigma which is about the 0.5 percentile. -right so far - and then burbled > The inner fences are selected to give a false positive rate of about 1 > in 1000. > > I suppose that if we take into account the Unwritten Rule of Antique > Statistics that all data sets have 30 elements, this *does* give > a "p-value" of (1-e)*30*0.001 = 5% which is obviously wrong. The false positive rate is about 1 in 100 and my fanciful 5% calculation is unsalvageable. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Boston Globe: MCAS results show weakness in teens' grasp of
Donald Burrill wrote: > > On Tue, 28 Aug 2001, Dennis Roberts wrote in part: > > > however ... the "flagging" of "outliers" is totally arbitrary ... i > > see no rationale for saying that if a data point is 1.5 IQRs away from > > some point ... that there is something significant about that > > If the data are normally distributed (or even approximately so, what > seems to be called "empirically distributed" these days), the 3rd > quartile + 1.5 IQR locates a point 2.0 std. devs. above the mean; > symmetrically, the 1st quartile minus 1.5 IQR gets you 2.0 SDs below the > mean. Close enough to the central 95% of the distribution, for the > precision of the "1.5". Er, no. Q1 ~ mu - 2/3 sigma Q3 ~ mu + 2/3 sigma 1 IQR ~ 4/3 sigma 1.5 IQR ~ 2 sigma inner fence ~ mu +- 2 2/3 sigma which is about the 0.5 percentile. The inner fences are selected to give a false positive rate of about 1 in 1000. I suppose that if we take into account the Unwritten Rule of Antique Statistics that all data sets have 30 elements, this *does* give a "p-value" of (1-e)*30*0.001 = 5% -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Boston Globe: MCAS results show weakness in teens' grasp of
Dennis Roberts wrote: > of course, research eons ago has shown that test performance is optimized > ... by having items in the order of easy to difficult ... IF there is a > time limit where some examinees have to push to get finished > > now that's a thought ... maybe if the items WERE ordered that way ... some > of that large % that seem to be failing ... would gain an item or two in > their score and pass!!! what a simple thing to do to make the students in > mass. look better! and mass. education! It seems to me that the "advantage" given by putting the problems in approximate order of difficulty is legitimate - much like the "advantage" given by having the questions printed in a legible font, by not interrupting the test several times for corrections and random information, or by not scheduling a fire drill during the test. If indeed the scores are being reduced by hiding the easy questions among the harder ones, then I would say yes, this is a defect of the current system, and should be changed. It may be that the questions themselves ought to be more difficult; but the difficulty ought to be intrinsic to the questions, not an artifact of the test format. What is at issue here is essentially signal-to-noise ratio. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Boston Globe: MCAS results show weakness in teens' grasp of
I wrote: > > > An obvious approach that would seem to give the advantages hoped for > >from the focussed test without the disadvantages would be just to group > >questions in the original test in roughly increasing order of > >difficulty. which, I think, answers Dennis' question. I wouldn't use sad faces for the hard ones - maybe exclamation marks or stars. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Boston Globe: MCAS results show weakness in teens' grasp of
> The focused test eliminates questions that would enable students to score A's, > B's, and C's. All they get is another chance to score a D rather than an F. > Implicit in the very concept of the focused test is the idea that a student who > fails the standard MCAS test cannot be more than a D student. Such students > will be accommodated with a special test on which they cannot score better than > a D: MCAS for Dummies. The cynicism of the proposal is almost beyond belief ... > The board is telling every 10th-grader in Massachusetts: "It's not important to > pass the MCAS on the first try. Not only will we give you four more chances, we > will offer you an easier test if you fail the first time. That's your prize for > losing." The booby prize turns out to be first prize. The focussed test isn't an entirely bad idea; it does allow a genuine D student to avoid getting blown out of the water by questions intended to discriminate between A and B students. However, it seems like a very poor way to carry the idea out. An obvious approach that would seem to give the advantages hoped for from the focussed test without the disadvantages would be just to group questions in the original test in roughly increasing order of difficulty. One might (I'm not so sure that this would be a good idea) put between each group a rubric along the lines of === PROGRESS CHECKPOINT 1 If you have got the right answers to 15 of the preceding 20 questions you have already passed (50%). If you have got the right answers to 18 of the preceding 20 questions you have already got at least a C- (60%). The questions below are mostly more advanced. Right answers to any of them will raise your numerical mark further and may raise your letter grade. KEEP ON GOING! === Alternatively one could use icons - say Happy Faces - to identify a set of recommended easier questions that nobody should quit without attempting, if there was a reason to use the order to encode something else. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
OT: SCAM WARNING Re: urgent response needed
"Gordon D. Pusch" wrote: > > [EMAIL PROTECTED] (Chike Ubah) writes: > > > ATTENTION: > > Dear Sir, > > This letter might surprise you because; we have not > > met neither in person nor by correspondence. But I > > believe it is one day that you get know somebody > > either in physical or through correspondence. > [ Rest deleted ] > > WARNING: This is a scam... Of course it is. OTOH, as almost every version of this that I've seen proposes partnership in highly illegal activities, it could be argued that there is no point protecting those who are dishonest enough to bite the hook. OTGH, anybody daft or ill-informed enough to think - by now - that this was for real might also be suspected of being unaware of the illegality of the proposed actions (or, indeed, of the color of the sky on the planet occupied by most of us); so perhaps solicitude for the gravely clue-challenged might justify a warning after all. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: definition of " metric" as a noun
> As for the other examples, 'professional' as a noun was good enough > for Dickens. Milton (1671) uses 'academic' as a noun. And Shakespeare, "mechanical". -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Venn diagram program?
> Thanks Alan for the constructive reply. The others so far remind me > of a Monty Python routine. Let me guess - the one in which the film producer fires everybody who comments on his idea? In which case, I hereby withdraw my comment that a Venn diagram might not be the ideal way to communicate this data. My only comment on the matter is now "Splunge". -Robert Dawson PS: For the perplexed: see http://www.montypython.net/scripts/20thvole.php = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Venn diagram program?
John Uebersax wrote: > > Can anyone suggest a standalone Windows (or DOS) that produces > publication quality Venn diagrams? ... > The diagram should show the area of each circle as proportional its N, > and the overlap area as proprotional to the number of cases in both > groups. Is this advisable? It would seem that the different shapes of the intersection and difference sections violate one of the standard rules for good plotting. I would guess that most untrained viewers would exaggerate the area of the lens and underestimate that of the crescent. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Presenting results of categorical data?
Thom Baguley wrote:
however, I think the
> defence of convenience samples can be stronger than this. Unless we
> have reason to believe that a sample is biased in such a way as to
> generate our pattern of results a convenience sample is just as good
> evidence as a (hypothetical) random sample.
This gets onto tricky philosophical ground. On the one hand, the
"argument from ignorance" ("if we don't have reason to believe
otherwise...") can be abused dreadfully. On the other hand it
underlies everything we do; it would seem to be perfectly rational to
start your car in the morning without looking under it on the grounds
that you don't have reason to believe that anybody wired a bomb to the
ignition.
Ideally I'd like positive reasons to believe that the sample WILL be
like the population to which inference is extended; but I'm not sure of
the extent to which this process can be made either formal or foolproof,
while still practical.
Random sampling manages "foolproof" but not "practical" in many
circumstances. I'm not sure to what extent it is formal, either; random
numbers work fine as abstract mathematical constructs but the
definitions don't tell you where in the real world to look for these
things.
Most so-called random numbers are pseudorandom and work just fine.
Whether mechanical methods like coin-tossing are truly random seems to
depend on your interpretation of quantum mechanics; again, if somebody
should come up with a souped-up hidden-variable theory that gets around
Bell's inequality next week, does statistical practice become invalid?
I think not.
Ultimately scientists still seem to need common sense.
=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=
Re: Presenting results of categorical data?
Jon Cryer wrote:
>
> I do not see how (probabilistic) inference is appropriate here at all.
Oh, it never is (strictly), outside of a few industrial
applications. Nobody ever took a random equal-probability sample from
all turnips, all cancer patients, all batches of stainless steel, all
white mice, or all squirrels. However, there are good common-sense
reasons to believe that *some* convenience samples will act enough like
true random samples to be useful. Otherwise we could stop teaching
statistics to biology and psychology students.
Indeed, it could be argued that pure science *only* makes inferences
about populations for which no sampling frame can be constructed. Yes,
you may be as random as you like about choosing a control group of 50
from your 100 white mice. But unless you title your paper
"The effect of resublimated thiotimoline on the reaction times of one
particular population of 100 white mice (now deceased)"
you are making inferences about a *different* population - that of all
white mice - and the only reason to extend the inference to that
population (rather to that of termites, or light bulbs) is because your
original 100 are a convenience sample from the population of all white
mice.
> I assume that _all_ employees are rated. There is no sampling, random
> or otherwise.
True. However, it seems reasonable to consider such a data set as a
pseudo-random sample from the "Platonic" set of "all potential employees
under this system" and interpret the inference in terms of the existence
of a pattern.
For instance: if I determine the mean height of all named peaks in the
Rockies whose names start with A,B,...,M, and compare that with the mean
height of all named peaks whose names start with N,...,Z, presumably one
mean will be greater. However, this is presumably a "fact without a
reason"; and many of us would place moderate wagers, if the odds were
right, on the difference between the means behaving much like the
difference between two random samples from the same population.
Moreover, should it turn out that this was *not* the case, no doubt you
(like most of us) would ask "why?" (perhaps the most prominent peaks had
been named for saints ("S")? Perhaps a couple super-high peaks
dominated and invalidated distributional assumptions?)
Now, if the same comparison were done between the Rockies and the
Himalayas, we would *not* be surprised to see a difference bigger than
the t distribution might predict. (Again - these are not random
samples.) And why?
"Because the Himalayas are higher than the Rockies." (Duh!)
Ah, but (let's say) the A-M Rockies are bigger than the N-Z Rockies.
"Ah, but that's just chance, there's no real pattern there."
Precisely.
To summarize: In most disciplines, there are no true random samples.
There are better and worse convenience samples
In most disciplines, there are no sampling frames. There are
populations, usually to some extent abstract or at least unlistable.
Inference is ultimately not about populations, but about patterns. A
property of an entire well-defined population is one example of a
pattern; there are others.
It behooves us to accept this and work with it, rather than to delude
ourselves that we sometimes have a true random sample.
-Robert Dawson
=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=
Re: Presenting results of categorical data?
"Silvert, Henry" wrote: > > I would like to add that with this kind of data [three-level ordinal] > we use the median instead of the average. Might I suggest that *neither* is appropriate for most purposes? In many ways, three-level ordinal data is like dichotomous data - though there are a couple critical differences. Nobody would use the median (which essentially coincides with the mode) for dichotomous data unless thay had a very specific reason for wanting that specific bit of information (and I use the word "bit" in its technical sense.) By contrast, the mean (=proportion) is a lossless summary of the data up to permutation (and hence a sufficient statistic for any inference that assumes an IID model) - about as good as you can get. With three levels, the mean is of course hopelessly uninterpretable without some way to establish the relative distances between the levels. However, the median is still almost information-free (total calorie content per 100-gram serving <= log_2(3) < 2 bits). I would suggest that unless there is an extremely good reason to summarize the data as ONE number, three-level ordinal data should be presented as a frequency table. Technically one row could be omitted but there is no strong reason to do so. "What about inference?" Well, one could create various nice modifications on a confidence interval; most informative might be a confidence (or likelihood) region within a homogeneous triangle plot, but a double confidence interval for the two cutoff points would be easier. As for testing - first decide what your question is. If it *is* really "are the employees in state X better than those in state Y?" you must then decide what you mean by "better". *Do* you give any weight to the number of "exceeded expectations" responses? Do you find 30-40-30 to be better than 20-60-20, equal, or worse? What about 20-50-30? If you can answer all questions of this type, by the way, you may be ready to establish a scale to convert your data to ratio. If you can't, you will have to forego your hopes of One Big Hypothesis Test. I do realize that we have a cultural belief in total ordering and single parameters, and we tend to take things like stock-market and cost-of-living indices, championships and MVP awards, and quality- of- living indices, more seriously than we should. We tend to prefer events not to end in draws; sports that can end in a draw tend to have (sometimes rather silly) tiebreaking mechanisms added to them. Even in sports (chess, boxing) in which the outcomes of (one-on-one) events are known to be sometimes intransitive, we insist on "finding a champion". But perhaps the statistical community ought to take the lead in opposing this bad habit! To say that "75% of all respondents ranked Ohio employees as having 'Met Expectations' or 'Exceeded Expectations.' ", as a single measure, is not a great deal better than taking the mean in terms of information content *or* arbitrariness. Pooling two levels and taking the proportion is just taking the mean with a 0-1-1 coding. It says, in effect, that we will consider (Exceed - Meet)/(Meet - Fail) = 0 while taking the mean with a 0-1-2 coding says that we will consider (Exceed - Meet)/(Meet - Fail) = 1. One is no less arbitrary than the other. (An amusing analogy can be drawn with regression, when users of OLS regression, implicitly assuming all the variation to be in the dependent variable, sometimes criticise the users of neutral regression for being "arbitrary" in assuming the variance to be equally divided.) -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: likert scale items
Rich Ulrich is correct that a dichotomy is interval, at least as far as what arithmetic it is permissible to perform on it. It is not so clear that it is interval in terms of what operations are NOT trivial to perform on it. If I have numerically-coded ordinal data I may recode using the square roots without distorting anything. If I have normal interval data and so recode I am looking at something different (tho' possibly useful.) OK, now, which of these is more like dichotomous data? It is also not clear that it is interval - or even ordinal - in terms of what summaries are useful or appropriate. "Standard" interval data is usually usefully described by mean and standard deviation; despite some author's claims I do not think the SD of dichotomous data is very useful, and even the mean's interpretation is only simple for 0-1 coding. "Standard" ordinal data is usually usefully described by (say) five-number summary. This is a very, very crude description of dichotomous data. By contrast, a good argument can be made for describing categorical data, as a first and second approximation, by (a) the mode and (b) the proportion of the data in (or not in) the mode. [Idea: Modeis to P(not in mode) as meanis to standard deviation as median is to MAD; by which I mean that each measure of location can be defined as minimizing the related measure of spread. EG: sum(y-x_i)^2 is minimized for y=xbar.] And this gives a perfect description of a dichotomy. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: likert scale items - why not PCA?
Magenta wrote: > > "Robert J. MacG. Dawson" <[EMAIL PROTECTED]> wrote in message > [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > > Can you do factor ananlysis, then? Probably yes. ... > > Why a factor analysis and not a principal components analysis? Dunno. But the interesting part of the question was about the non-interval data, not FA vs PCA. -RD = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: likert scale items
Teen Assessment Project wrote: > > I am using a measure with likert scale items. Original psychometrics > for the measure > included factor analysis to reduce the 100 variables to 20 composites. > However, since the variables are not interval, shouldn't non-parametic > tests be done to determine group differences (by gender, age, income) on > the variables? Can I still use the composites...was it appropriate to > do the original factor analysis on ordinal data? When you "cast" ordinal data to interval as you have done, there are at least two questions that must be considered: (1) Are the sampling distributions approximately valid for the tests that you have done? (2) Can you give any valid interpretation to the outcome? The first question seems to be the one people worry about most; but if you have enough data to do factor analysis on 100 variables (which means WELL over a hundred independent samples, probably several hundred), you almost certainly have enough data that you can put your faith in the Central Limit Theorem. That is, if your test says that the mean of Factor A is higher for Group 1 at a significance level of 2.3%, then that is pretty much what you would get from a test based on the *true* sampling distribution, and not an artifact of your choice of tests. OK, so what? Having rejected H-nought at yadda yadda, you are entitled to state that the mean Likert score is greater than something or other, (or whatever.) However, that's not what the punters want to know. They (and you) want to know if issuing teddy bears to the residents of halfway houses reduces aggression, (again, or whatever.) *There* *is* *no* *way* to compute a "mean" of actual levels of aggression. (Add two levels of aggression and you get a brawl, which is dangerous to divide by 2.) The true advantage of tests for the median here has nothing to do with the sampling distribution; rather, it stems from the fact that one *can* assign some sort of value to "median level of aggression" independent of the Likert scale and one cannot do the same for "mean level of aggression" or whatever. By the way - this does NOT necessarily require the use of nonparametrics. While it is true that the sign test (for instance) is a test for the median in all cases: (a) not all nonparametric tests are tests for the median. The Wilcoxon-Mann-Whitney test is NOT in general a test for the median. It is if the two populations (a) have the same shape and spread or (b) are symmetric. In these cases it is also a test for the mean! In general (even for Behrens-Fisher alternatives [in which spread and location, but not shape, can vary] it is not a test for *any* measure of location. I can send a paper proving this to anybody interested.) (b) not all tests for the median are nonparametric. For symmetric distributions with well-defined median (ie, no hole in the middle) the mean IS the median, and within such families the t-test is a test for the median. Not only that- it's often more powerful than the nonparametric tests. Can you do factor ananlysis, then? Probably yes. If the responses approximately line up in subspaces, that's a mathematical fact; you can do arithmetic on the responses. Once you've isolated the factors, though, it would seem reasonable to make inferences about the median, not about the mean. BUT, if the factor score distributions are roughly symmetric, this might be done using a t test! (A caution: the t test is less robust to outliers than the nonparametrics. OTOH, Likert data is usually free of univariate outliers.) -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: EdStat: Probabilistic inference in resampling?
Herman Rubin wrote: > > I consider the Fisherian one to be the only relevant one. > In fact, I do not think it goes far enough; at best, > probability is a property of the real world like length > and mass. On the contrary: length and mass are abstractions that approximately describe certain aspects of the real world (quantum mechanics demonstrates this.) They can also both be used, advan- tageously, to describe other aspects of the real world other than those that we normally associate with them (for instance, dimensional analysis makes good use of the fact that the natural dimension of capacitance is length...) Probability works like that too. Mathematically probability is very simple and paradox-free; it "exists" only in the sense that 24-dimensional quaternionic space or the Monster Group "exist". Various real-world things behave in roughly the same way: both long-run frequencies (the frequentist application of probability theory to the real world) and degrees of belief subject to certain mental disciplines that individuals may or may not adhere to (the Bayesian application). (It is undoubtedly true that in the real world a person may well "believe" incompatible things. The word "believe" is not the3 property of the statistical community and we cannot say "oh no you don't" to such a person. However, those whose degrees of belief in related events do *not* fit the Bayesian yoga and who are prepared to bet when the odds give them an edge according to their own beliefs can be reliably (in the long run) relieved of their stakes. This gives some objective basis to the idea of consistent belief systems.) -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Book on medical/epidemiological studies?
A friend is looking for a book on designing medical and epidemiological studies. What do you recommend? -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Bayesian analyses in education
KKARMA wrote:
>
> As a teacher of research methodology in (music) education I am
> interested in the relation between traditional statistics and the
> bayesian approach. Bayesians claim that their approach is superior
> compared with the traditional, for instance because it does not assume
> normal distributions,
Sometimes it does; and sometimes "classical" stats doesn't.
>is intuitively understandable,
Some bits are. Perhaps it's more accurate to say that while the
premises of Bayesian statistics are more intuitive at first glance, and
the conclusions come closer to what people intuitively want to know
("what is the probability that this is correct?") Bayesian stats has
its own thorny philosophical problems.
Frequentist stats says that only statements about the outcome of random
variables have "probability"; it is legitimate to say that "the
probability that this die shows a 6 on this roll is 1/6" but not (unless
the die itself was drawn at random from a well-defined collection)
"there is an 80% probability that this die rolls 6 more often than 1".
Bayesians do allow the broader use of probability to describe levels of
belief about something that was not generated by a well-defined sampling
operation, but the cost of this is that "the pump must be primed" with a
"prior probability" representing your level of belief before the
observations, and this is necessarily subjective.
The cost is not as great as it appears, because as the data accumulates
the impact of the prior becomes less and less; that is, rational
observers with initially different beliefs come to more or less agree
after observing the data.
The justification for this is that a Bayesian interpretation of an
opinion poll can actually be "The probability that the Garden Party
would get more than 40% of the votes in this election is x% [if it were
held today and voting patterns matched polling response patterns]"
whereas - despite the fact that this is intuitively the answer to the
question people *want* to ask- frequentist stats cannot.
The frequentist can only assign probabilities to samples from
well-defined populations. So the frequentist analysis of the same poll
might be "IF the Garden Party would have exactly 40% popular support
[if the election were held today and polling response patterns matched
voting patterns] the probability of getting this result or one less
favorable to the Garden Party in an opinion poll done in this way would
be y%."
> works with small samples,
To some extent; and so does frequentist stats, to some extent. Both
tend to be inconclusive with small samples, and to get some of whatever
power they have from assumptions that the data cannot justify. That's
the way the universe works: you want answers, first get enough data.
> If this is so, why is it so rare in educational research? Are there some
> hidden flaws in the approach or are the researchers just ignorant?
(1) Propagation delay. What statisticians are writing about in
theoretical journals today will be used by statisticians in their
practical work in a few years. It'll be in upper-level stats textbooks
for stats majors in a decade. Maybe in two decades significant numbers
of social science PhD's will start to hear about it; maybe another ten
years later somebody will be bold enough to put it in an applied stats
textbook; at that point it becomes well enough known to be used widely.
Maybe.
(2) Encapsulation. The philosophical complications of the frequentist
method are well-hidden for most users inside phrases such as "confidence
interval" and "significance level". You can construct a 95% confidence
interval correctly while believing that this guarantees a 95%
probability that this particular interval contains the true value (which
is not so); and you can even state this in your paper and many referees
and editors will let it pass. Similarly, you can do a hypothesis test at
the 5% significance level while believing that 5% is "the probability
that your data are wrong" (it isn't.). If people were required to
truly understand hypothesis tests and confidence intervals before using
them there might be more impetus for change.
Note: This is NOT entirely a valid argument for change, any more than
saying that "if people were required to understand the workings of their
vehicles there would be a lot more bikes on the road and a lot fewer
cars" is. On the other hand, if there were no mechanics, it might be.
(3) Standardization: There are genuine advantages to everybody singing
from the same hymnbook, which tends to lead to change being slow.
-Robert Dawson
=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=
Re: C.I. for a percentage
Chris wrote: > > I know how to construct a confidence interval for a percentage when > there is a binomial (# of good units / # of total units) distribution. > How is the C.I. constructed if I have data such as the % of a liquid > evaporated for each of a number of batches and I want to know let's > say the limits of 99% of the population of the batches? Each batch > has a %, say x kg out of y kg. I don't think a kg can be considered a > "unit" (or can it?) The fact that the data involve "percentages" here is a red herring. A kilogram cannot be considered a unit in your sense, as its size is just a convention; and moreover, liquids do not evaporate by a Poisson/binomial process in which each individual gallon/kilo/firkin has a certain chance of evaporating *pop!* in one go, independent of the rest. For large samples (number of batches) a normal model with t test/CI would probably be appropriate. How large "large" is depends on the distribution. For small samples detailed mathematical models of evaporation might be needed - is the randomness due to temperature fluctuations? purity fluctuations? cross section of the tank as a function of depth? windspeed? somebody leaving the top off sometimes? Depending on all that, you might be able to get away with a normal model; or you might need to transform or use a nonparametric method or something. Indeed, it is possible that a CI for percent evaporation might not even be a good summary of the observations in some of these cases - say that the evaporation is either 0.05% if the lid is left on or 4% per day if it's left off. You'd have a mixed model with a large spike near 0 and along tail; "average percent loss" would be a bad description of reality. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What does it mean?
"J. Williams" wrote: > > On Mon, 09 Jul 2001 12:15:25 GMT, Jan Sjogren > <[EMAIL PROTECTED]> wrote: > > >Hi there! > > > >I wonder what these things means: > > > >SST > >SSM > >SSE > >SSR > >MSR > >MSE > > > >Thanks, > > > >Janne > > > Are these statistical acronyms you want defined? SSt, for example, > could be total sum of squares and SSe could be sum of squares error. > SST could denote the Concorde aircraft type or a Russian TU 144. Oh, come *on*. Somebody posts to a stats mailing list with a list of closely related stats acronyms and we start winding them up with the suggestion that SST might be an airplane? What other plausible model would explain that list of acronyms as a set? Jeeesh. Janne: These are acronyms used in ANOVA and regression to describe various sources of variation in the data. SS? - Sum of Squares, MS? = Mean Square. T = Total - the total variation shown by the data. M = Model (roughly synonymous with Tr , "treatment" or R, "regression") - the variation that can be explained by the explanatory variables in the fitted model. "Regression" usually implies a numerical predictor, "Treatment" a categorical one, but there is less difference between regression and ANOVA than might at first appear. E = Error - the variation *not* explained by the model. It may be "error" in the sense of observational sloppiness, actual variation in what's being observed, or many other things; it's a catchall term. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: individual item analysis
Jay Warner wrote: > > If the distribution is monomodal, not bounded (one peak, tails on both > sides), and within some rational range of a Normal, then a transformation > can be performed to make it look 'Normal.' See the Johnson transformation > in some commercial stat packages. > > The next question is why would you bother? Looks pretty, feels more > comfortable to one with experience only with Normal dist.'s, but once you > know the transformation there is no greater or less amount of information in > it, as I understand such things. One good reason: knowing the transformation that symmetrizes the distribution can give a useful description of the distribution. For instance, if you know that a log transformation renders a data set approximately normal, then you can conjecture that the process that generates the variation within it is essentially multiplicative. *No* transformation can add information content to a data set; but that is not what transformations are for. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: A question
Alan McLean wrote: > > Hi to all. > > Can anyone tell me what is the distribution of the ratio of sample > variances when the ratio of population vriances is not 1, but some > specified other number? *If* the population distributions are normal (and this is not a robust assumption - in other words, if it's moderately wrong you are *not* safe from error) it's just a scaled F distribution. If X has variance a^2, Y had variance b^2, then (b^2/a^2) s^2_X/s^2_Y = s^2_(X/a)/s^2_(Y/b) ~ F . -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Artifacts in stats: (Was Student's t vs. z tests)
> Alan McLean wrote: The p value is a direct measure of 'strength of evidence'. and Lise DeShea responded: > > I disagree. The p-value may be small when a > study has enormous power yet a small effect size. A p-value by itself doesn't say much. I don't think there's actually a contradiction here, provided that "strenth of evidence" [against the null hypothesis] is not misunderstood to mean "strength of evidence for the conclusion you are trying to draw", this latter rarely being the literal denial of the null hypothesis. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Student's t vs. z tests
dennis roberts wrote: > > the fundamental issue here is ... is it reasonably to expect ... that when > you are making some inference about a population mean ... that you will > KNOW the variance in the population? No, Dennis, of course it isn't - at least in the social sciences and usually elsewhere as well. That's why I don't recommend teaching this (recall my comments about "dangerous scaffolding") to the average life-sciences student who needs to know how to use the test and what it _means_, but not the theory behind it. In the case of the student with some mathematical background, who may actually need to do something theoretical with the distribution one day (and may actually have the ability to do so) I would introduce t by way of Z. A rough guide; If this group of students know what a maximum-likelihood estimator is, and have been or will be expected to derive, from first principles, a hypothesis test or confidence interval for (say) a singleton sample from an exponential distribution, then they ought to be introduced by way of Z. If not, then: (a) don't do it at all, or (b) put your chalk down and talk your way through it as an Interesting Historical Anecdote without giving them anything to write down. Draw a few pictures if you must. Or (c) give them a handout with "DO NOT USE THIS TECHNIQUE!" written on it in big letters. (I've tried all four approaches, as well as the wrong one.) -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Student's t vs. z tests
Jon Cryer wrote: > > These examples come the closest I have seen to having a known variance. > However, often measuring instruments, such as micrometers, quote their > accuracy as a percentage of the size of the measurement. Thus, if you > don't know the mean you also don't know the variance. You do if you log-transform... -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Student's t vs. z tests
Paul Swank wrote: > > However, rather than do that why not right on to F? Why do t at all when you can do >anything with F that t can do plus a whole lot more? Because the mean, normalized using the hypothesized mean and the observed standard deviation, has a t distribution and not an F distribution. I am aware that the two are algebraically related,(and simply) but trying to get through statistics with only one table (or only one menu item on your stats software) seems pointless - like trying to do all your logic with NAND operations just because you can. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Eight Features of an Ideal Intro Stat Course
> If the original cold fusion researchers had used proper statisti- > cal methods for detecting relationships (i.e., statistical tests, > a proper taking account of negative results, and experimental de- > sign considerations), it seems likely that the repeated high or > borderline p-values would have quickly alerted them to the fact > that that the sought-after relationships between the variables > were probably not present. Or if the relationships WERE present, > they were not properly detectable with the current experimental > methods, and thus further "more powerful" research must be per- > formed before a responsible positive conclusion could be drawn. > This might have saved the original researchers considerable em- > barrassment and might have saved the physical science community > substantial costs. Surely here the problem was not poor statistical procedures per se but poor experimental design. Individual results were claimed that were completely at odds with existing theory; if these were not real it was because the experiment was designed badly, and perhaps because insufficient control runs (omitting one or more supposedly critical components) were performed. Even a single run in which unexplained energy is released in significant amounts would be important if it were properly designed; whether or not one could prove anything about the mean over a dozen runs would be irrelevant. Statistical analysis becomes important later on when one is fine-tuning the process. Conversely, a process in which many runs and statistical anaysis were needed to show the existence of an effect would be likely to remain a laboratory curiosity. (Given the world's limited supply of palladium, the process as described seemed like a bad joke on the part of Nature anyway; what use to the world is perpetual free power in strictly limited quantities?) -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: 95% confidence interval
Radford Neal wrote: > As another poster has said, one reason is technical convenience. A > more fundamental reason, though, is that the median is probably not > the best thing to look at, whatever you might have been taught. What > to look at depends not just on the shape of the distribution, but also > on what your purpose is. Ask yourself whether there are very many > purposes for which it would make no difference if the upper 49% of the > incomes were doubled, leaving the median unchanged. If one assumes that that *could* happen, then no single summary statistic will be of much use - one is implicitly assuming a very general family of models. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: The meaning of the p value
Will Hopkins wrote: > > I accept that there are unusual cases where the null hypothesis has a > finite probability of being be true, but I still can't see the point in > hypothesizing a null, not in biomedical disciplines, anyway. > > If only we could replace the p value with a probability that the true > effect is negative (or has the opposite sign to the observed effect). The > easiest way would be to insist on one-tailed tests for everything. Then > the p value would mean exactly that. An example of two wrongs making a right. No, a one-tailed test doesn't work; it is still computed using the null value. To find what you want you need Bayesian techniques... but then (if your prior distribution is valid) you can answer the question you *really* wanted to answer - "what is the probability that the effect exists?" Or even "what is the distribution of the parameter value?" -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: 95% confidence interval
Radford Neal wrote: > > In article <[EMAIL PROTECTED]>, > James Ankeny <[EMAIL PROTECTED]> wrote: > > >... if the distribution is > >heavily skewed to the right, say like income, why do we want an interval for > >the population mean, when we are taught that the median is a better measure > >of central tendency for skewed distributions? > > As another poster has said, one reason is technical convenience. A > more fundamental reason, though, is that the median is probably not > the best thing to look at, whatever you might have been taught. But that's not what he said - he said it was _better_ than the mean, as a criticism of using the mean. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: 95% confidence interval
James Ankeny wrote: > > Hello, >I am currently taking a first course in statistics, and I was hoping that > perhaps someone might be kind enough to answer a question for me. I > understand that, while a quantitative variable may not be normally > distributed, we may calculate the mean of the sample, and use facts about > the Central Limit Theorem, to form a 95% confidence interval for the > population mean. As far as I know, this means that in 95/100 samples, the > interval will contain the true population mean. This seems very useful at > first, but then something begins to confuse me. Yes, we have an interval > that may contain the true population mean, but ... if the distribution is > heavily skewed to the right, say like income, why do we want an interval for > the population mean, when we are taught that the median is a better measure > of central tendency for skewed distributions? Excellent question. The answer is, we often don't; and we can instead transform (say by logs) and get an interval for a transformed mean (here the geometric mean.) Or we can use a sign interval (nonparametric) and get an interval estimate for the median. . There is just one more > thing I would like to get off my chest. My textbook talks about simple > random sampling, where you can specify the probability of a sample being > selected from the population. Yet, there are examples in the book which deal > with conceptual populations, such as the set of all cars of a particular > model which may be manufactured in the future. Suppose you have a sample of > several of these autos, and you want to find a 95% confidence interval for > mean miles/gallon. How is this an SRS when you can't specify the probability > of a sample being selected, because the population is conceptual? It isn't one, but it acts in much the same way. In practice, the SRS model and real samples have practically no intersection. One or more of of the following almost always happens: *Your population is theoretical (eg, testing to see whether whatever difference may exist between the lifetimes of American Presidents and British monarchs are statistically significant, using *all* the ages as a pseudosample) *Your population is somewhat vaguely defined *Your population is not all available for listing or samping (a "random" sample of wild squirrels in Maine) *You want to make inferences about all human beings based on the Psych 1000 subject pool, or any other "convenience" group. Fortunately, you *may* be able to draw valid conclusions even in these circumstances if you use common sense, though the statistics will be dubious. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: The meaning of the p value
Bruce Weaver wrote: > Suppose you were conducting a test with someone who claimed to have ESP, > such that they were able to predict accurately which card would be turned > up next from a well-shuffled deck of cards. The null hypothesis, I think, > would be that the person does not have ESP. Is this null false? Technically, the null hypothesis is that P(card is predicted correctly) = 1/52 - it is a statement about parameter values. Thus, any bias, no matter how slight, affecting this, would make Ho false - whether the subject had ESP or no. For instance, if the shuffling method tended to make a card slightly less likely to come up twice in a row than one would expect, *even by a few parts in a million*, and if the subject avoided such guesses, then Ho would indeed be false. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Excel Graphics
Shareef Siddeek wrote: > > Then, what is the use of EXCEL? EXCELlent question... Joking apart, it can be a useful tool for preparing a downloaded dataset for loading into a statistics program. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: 280 or not
Gene Gallagher wrote: > Imagine this as an MCAS question: > > A scientist has two thermometers, one Fahrenheit and the other Celsius. > When the temperature is less than 8.4 degrees, the scientist > uses the Celsius scale. When the temperature is more than > 8.4 degrees, he uses the Fahrenheit scale. In 1998, he took 40 > measurements of the temperature. In the two following years, he > took 40 more measurements of the temperature each year. > The mean temperature was one degree less in 1998 than the mean > of the 1999 and 2000 temperatures. Should the scientist: > a) Conclude that the temperature is getting colder > b) Conclude that the temperature is about the same > c) Conclude that the temperature is getting warmer > d) Not conclude anything since it is meaningless to average >degrees Farhenheit and degrees Celsius in this way. Interesting. If it's 8.4 degrees Fahrenheit at which he switches, no conclusion could be drawn for *any* sample size because the recorded value would not be a monotonic function of actual temperature. ; if the change takes place at 8.4C the sample size might be insufficient, but as the scale is monotone (the MCAS score is) valid conclusions could be drawn from a large enough sample (the ubiquitous central limit theorem!) - with the understanding that they were not in general conclusions about the arithmetic mean temperature, but about another measire of location. However, we have to understand that in the absence of a well-defined ONE-parameter family of alternatives (eg, shift, scale, etc) or a class of distributions such as the symmetric distributions for which "location" is hard to define in any other way, the assumption that "score increasing" must mean "arithmetic mean of scores increasing" is arbitrary. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: I dont know what the question is asking me? Please HELP!
> > Here is the question: > > > > Car A: 27.930.430.631.431.7 > > > > (a) If the manufactureres of car A want to advertise that their car > > performed best in this test, which of the "average" discussed so far in > > this chapter could they use to substantiate this claim? Surely the question also contains a similar set of numbers for one or more other cars? -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: MA MCAS statistical fallacy, rather OT
"P.G.Hamer" wrote: > > dennis roberts wrote: > > > there just is no good way to argue against the original choice C ... IN THE > > CONTEXT OF THE STEM OF THE QUESTION > > I am reminded of the joke article that contains many `politically incorrect' > answers to the exam question "given a barometer how do you measure > the height of a tower". > > A point I only realised recently is that many of these spoof answers could > give a more accurate answer than the `textbook' method. > > Peter > > The first few that I remember. > > 1) Drop the barameter and time its fall. > > 2) Tie it to a long piece of string and use it as a lead-line, measuring the > length of the string. > > 3) Tie it to a long piece of string and use it as a lead-line, measuring the > period of the resultant pendulum. My own favorite: tell the janitor that you will give him a barometer if he can tell you how tall the building is. I *presume* that "politically incorrect" above means "nonstandard", not "involving stereotypes of gender, religion, ethnicity, sexual orientation, or hair color". The idea of a whole subgenre of "Scottish Barometer Jokes" or "Blonde Barometer Jokes" is just too mind-boggling. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Excel Histograms
A histogram of a continuous variable is *not* primarily a graph of counts within a range, but of densities; that is, an approximation to a density function. Therefore, it is admissible to join or subdivide bars provided that the area of the bar, not the height, represents the joint count. When the ranges are equal, one can label the graph on one vertical edge with densities and on the other with counts. There might be situations in which the counts were truly more imortant than the density - for instance, "pass-fail" statistics based on numerical grades, or "too narrow-acceptable-too wide" for ball bearings. I would argue that in such a situation the count classifications were essentially categorical and should be represented by narrow, separated, bars to make this clear. -Robert Dawson = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
