Re: Applied analysis question
On 28 Feb 2002 07:37:16 -0800, [EMAIL PROTECTED] (Brad Anderson) wrote: > Rich Ulrich <[EMAIL PROTECTED]> wrote in message >news:<[EMAIL PROTECTED]>... > > On 27 Feb 2002 11:59:53 -0800, [EMAIL PROTECTED] (Brad Anderson) > > wrote: BA > > > > > > I have a continuous response variable that ranges from 0 to 750. I > > > only have 90 observations and 26 are at the lower limit of 0, which is > > > the modal category. The mean is about 60 and the median is 3; the > > > distribution is highly skewed, extremely kurtotic, etc. Obviously, > > > none of the power transformations are especially useful. The product > > [ snip, my own earlier comments ] BA > > I should have been more precise. It's technically a count variable > representing the number of times respondents report using dirty > needles/syringes after someone else had used them during the past 90 > days. Subjects were first asked to report the number of days they had > injected drugs, then the average number of times they injected on > injection days, and finally, on how many of those total times they had > used dirty needles/syringes. All of the subjects are injection drug > users, but not all use dirty needles. The reliability of reports near > 0 is likely much better than the reliability of estimates near 750. > Indeed, substantively, the difference between a 0 and 1 is much more > significant than the difference between a 749 and a 750--0 represents > no risk, 1 represents at least some risk, and high values--regardless > of the precision, represent high risk. Okay, here is a break for some comment by me. There are two immediate aims of analyses: to show that results are extreme enough that they don't happen by chance - statistical testing; and to characterize the results so that people can understand them - estimation. When the mean is 60 and the median is 3, giving report on averages, as if they were reports on central tendencies, is not going to help much with either aim. If you want to look at outcomes, you make groups (as you did) that seem somewhat homogeneous. 0 (if it is). 1. 2-3 eventually, your top group of 90+, which comes out to 'daily', seems reasonable as a top-end. Using groups ought to give you a robust test, whatever you are testing, unless those distinctions between 10 and 500 needle-sticks become important. Using groups also lets you inspect, in particular, the means for 0, 1, 2 and 3. I started thinking that the dimension is something like 'promiscuous use of dirty needles'; and I realized that an analogy to risky sex was not far wrong. Or, at any rate, doesn't seem far wrong to me. But your measure (the one that you mention, anyway) does not distinguish between 1 act each with 100 risky partners, and 100 acts with one. Anyway, one way to describe the groups is to have some experts place the reports of behaviors into 'risk-groups'. Or assign the risk scores. Assuming that those scores do describe your sample, without great non-normality, you should be able to use averages of risk-scores for a technical level of testing and reporting, and convert them back to the verbal anchor-descriptions in order to explain what they mean. [ ...Q about zero; kurtosis.] RU > > > > Categorizing the values into a few categories labeled, > > "none, almost none, " is one way to convert your scores. > > If those labels do make sense. > > Makes sense at the low end 0 risk. And at the high end I used 90+ > representing using a dirty needle/syringe once a day or more often. > The 2 middle categories were pretty arbitrary. [ snip, other procedures ] > One of the other posters asked about the appropriate error term--I > guess that lies at the heart of my inquiry. I have no idea what the > appropriate error term would be, and to best model such data. I often > deal with similar response variables that have distributions in which > observations are clustered at 1 or both ends of the continuum. In > most cases, these distributions are not even approximately unimodal > and a bit skewed--variables for which normalizing power > transformations make sense. Additionally, these typically aren't > outcomes that could be thought of as being generated by a gaussian > process. Can you describe them usefully? What is the shape of the behaviors that you observe or expect, corresponding to the drop-off of density near either extreme? > In some cases I think it makes sense to consider poisson and > generalizations of poisson processes although there is clearly much > greater between subject heterogeneity than assumed by a poisson > process. I estimated poission and negative binomial regression > models--there was compelling evidence t
Re: Applied analysis question
On 27 Feb 2002 17:16:26 -0800, [EMAIL PROTECTED] (Dennis Roberts) wrote: > i thought of a related data situation ...but at the opposite end > what if you were interested in the relationship between the time it takes > students to take a test AND their test score > > so, you have maybe 35 students in your 1 hour class that starts at 9AM ... > > you decide to note (by your watch) the time they turn in the test ... and > about 9:20 the first person turns it in ... then 9:35 the second 9:45 > the 3rd 9:47 the 4th ... and then, as you get to 10, when the time > limit is up ... the rest sort of come up to the desk at the same time > > for about 1/2 of the students, you can pretty accurately write down the > time ... but, as it gets closer to the time limit, you have more of a > (literal) rush and, at the end ... you probably put down the same time > on the last 8 students > > you could decide just to put the order of the answer sheet as it sits in > the pile ... or, you might collapse the set to 3 groupings ... quick turner > iners, middle time turner iners ... and slow turner iners BUT, this clouds > the data [ snip, rest] Looks to me like it might be reasonable to re-sort and re-score the speed as reciprocal, "questions per hour" -- instead of the original, hours per question. That emphasizes something you (perhaps) omitted: some tests at the end were incomplete. Also, Q/H accommodates that early test that was nearly blank. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Robust regression
On 1 Mar 2002 00:36:01 -0800, [EMAIL PROTECTED] (Alex Yu) wrote: > > I know that robust regression can downweight outliers. Should someone > apply robust regression when the data have skewed distributions but do not > have outliers? Regression assumptions require normality of residuals, but > not the normality of raw scores. So does it help at all to use robust > regression in this situation. Any help will be appreciated. Go ahead and do it if you want. If someone asks (or even if they don't), you can tell them that robust regression gives exactly the same result. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: help on factor analysis/non-normality
On 1 Mar 2002 04:51:42 -0800, [EMAIL PROTECTED] (Mobile Survey) wrote: > What do i do if I need to run a factor analysis and have non-normal > distribution for some of the items (indicators)? Does Principal > component analysis require the normality assumption. There is no problem of non-normality, except that it *implies* that decomposition *might* not give simple structures. Complications are more likely when covariances are high. What did you read, that you are trying to respond to? > Can I use GLS to > extract the factors and get over the problem of non-normality. Please > do give references if you are replying. > Thanks. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Applied analysis question
On 27 Feb 2002 14:14:44 -0800, [EMAIL PROTECTED] (Dennis Roberts) wrote: > At 04:11 PM 2/27/02 -0500, Rich Ulrich wrote: > > >Categorizing the values into a few categories labeled, > >"none, almost none, " is one way to convert your scores. > >If those labels do make sense. > well, if 750 has the same numerical sort of meaning as 0 (unit wise) ... in > terms of what is being measured then i would personally not think so SINCE, > the categories above 0 will encompass very wide ranges of possible values [ ... ] Frankly, the question is about meaning of numbers, and I would to ask it. I don't expect a bunch of zeros, with 3 as median, and values up to 750. Numbers like that *might* reflect, say, the amount of gold detected in some assays. Then, you want to know the handful of locations with numbers near 750. If any of the numbers at all are big enough to be interesting. Data like those are *not* apt to be congenial for taking means. And if 750 is meaningful, using ranks is apt to be nonsensical, too. In this example, the median was 3. Does *that* represent a useful interval from 0? - if so, *that* tells me scaling or scoring is probably not well-chosen. Is there a large range of 'meaning' between 0 and non-zero? Is there a range of meaning concealed within zero? "Zero children" as outcome of a marriage can reflect (a) a question being asked too early; (b) unfortunate happenstance; or (c) personal choice - categories, within 0, and none of them are necessarily a good 'interval' from the 1, 2, 3... answers. But that (further) depends on what questions are being asked. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Applied analysis question
On 27 Feb 2002 11:59:53 -0800, [EMAIL PROTECTED] (Brad Anderson) wrote: > I have a continuous response variable that ranges from 0 to 750. I > only have 90 observations and 26 are at the lower limit of 0, which is > the modal category. The mean is about 60 and the median is 3; the > distribution is highly skewed, extremely kurtotic, etc. Obviously, > none of the power transformations are especially useful. The product I guess it is 'continuous' except for having 26 ties at 0. I have to wonder how that set of scores arose, and also, what should a person guess about the *error* associated with those: Are the numbers near 750 measured with as much accuracy as the numbers near 3? How do zero scores arise? Is this truncation; the limit of practical measurement; or just what? "Extremely kurtotic," you say. That huge lump at 0 and skew is not consistent with what I think of as kurtosis, but I guess I have not paid attention to kurtosis at all, once I know that skewness is extraordinary. Categorizing the values into a few categories labeled, "none, almost none, " is one way to convert your scores. If those labels do make sense. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Means of semantic differential scales
> > > 2. Perhaps more likely, your boss may have learned > > (wrongly?) that parametric stats should not be done unless scales > > of measurement are at least interval in quality. > > I don't know if his objection was to parametric statistics per se, but he did > object to calculating means on these data, which he believes are only ordinal. > > > Search on google > > for people like John? Gaito and S.S. Stevens and for phrases like > > "scales of measurement" and "parametric statistics." > > Thanks. Will do. > Or, do an Advanced search with groups.google among the sci.stat.* groups for < Stevens, measurement >. I think that would find earlier discussions and some references. As I recall it, no one who pretended to know much would have sided with your boss. The firmness of Stevens's categories was strongly challenged by the early 1950s. In particular, there was Frederick Lord's ridiculing parable of the football jerseys. (Naturally, psychology departments taught the subject otherwise, for quite a while longer.) Conover, et al., took a lot of the glory out of 'nonparametric tests' by showing that you can't gain much from rank-order transformations, compared to any decent scaling. That was in an article of 1980 or thereabouts. I may have seen a 'research manual' dated as recent as 1985 that still favored using rank-statistics with Likert-scaled items. I am curious as to what more recent endorsements might exist, in any textbooks at all, or in papers by statisticians. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: REQ: Appendix A. of Radford Neal thesis: "Bayesian Learning for Neural Networks"
On Fri, 22 Feb 2002 18:00:16 +0100, Mark <[EMAIL PROTECTED]> wrote: > Hi, > > I'm CS student interested in Radford Neal thesis called "Bayesian > Learning for Neural Networks". I know that some years ago this thesis > was available for download from author's site, but nowadays there > isn't possible. I have searched it on Intenet so I have not known to > find it. Why not send personal e-mail and ask him? He has posted to the stat-groups within the last month, from Radford Neal ([EMAIL PROTECTED]) > My e-mail address is [EMAIL PROTECTED] Please remove the > "REMOVETHIS" string from the email address to get my real one. It's an > anty-spam measure. I apologize for any inconvenience that it causes to > you. - no inconvenience; I won't bother. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: odds vs probabilities
On 23 Feb 2002 06:49:58 -0800, [EMAIL PROTECTED] (Brad Branford) wrote: > hi, > > thanks. sorry if I posed the question poorly. actually, what I'm > looking for is an intuitive understanding of when to use odds and when > probabilities. I know that probs have a problem in that they don't > make multiplicative sense: for instance, assume I have a probability > of winning of 55%; if the likelihood of winning doubles, we have > absurd outcomes if expressed in terms of probabilities. > > thanks. Uh-oh. You have introduced a third technical term here. Likelihood doesn't match either probability or odds. There is a classical book by AWF Edwards by that title (Likelihood), which argues we should be using likelihood for all our inference: This makes some difference, though I am not sure of what and when and how much. Ps and likelihood compete in inference. Ps and ORs arise in descriptions of samples. Where Probability is constructed as an area under the pdf (often a 'tail area'), and OR is the ratio of two areas, the Likelihood is simply the height of the curve as evaluated at one point. Thus Probability and Odds are interchangeable, in just the way Ken describes. I think you only gain the 'intuitive understanding' by exposure to a whole gamut of examples, including the counter-examples on when (especially) probability does not work because it is misleading. For instance, in many contexts, 93% of a sample is not "nearly the same as" 99%, since the OR is 7.00, and that will matter. There is less reason to complain about P in place of ORs when the Ps are small -- where the arithmetic doesn't expose the fallacy, as with your "twice 55%". And P can be approximately correct in describing sample sizes when comparison-values are all between 25% and 75%, or 20-80. But generally ORs are more appropriate for statistical models. The drawback of ORs is that the public is apt to understand them less. A year or so ago, someone posted journal references with arguments *against* using ORs. I looked up a couple and did not find them impressive, but you can probably find them by searching sci.stat.*with groups.google.com . -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Evaluation of skating
On 19 Feb 2002 15:14:01 -0800, [EMAIL PROTECTED] (Trevor Bond) wrote: [ snip, much ] > affected who won the gold medal. In fact, Looney (1994, p. 156) > concluded: > "all of the judges with an Eastern block or communistic background > not only ranked Baiul better than expected, but ranked Kerrigan > worse. The same trend was seen for Western Block judges. They > ranked Baiul worse and Kerrigan better than expected. ... " Finding a difference is one thing. Drawing invidious conclusions is a gratuitous step for a statistician, isn't it? Hypothesize. Group A holds a bunch of their own, inter-community skating competitions. So does group B. This happens for many years. I find it wholly reasonable -- if not expected -- that 'community standards' might exist with some divergence. That's especially so when there were never any joint standards in the first place, and when one country has the outstanding professional dance (ballet) of the world, which is accorded much local respect. > When the > median of the expected ranks is determined, Kerrigan would be > declared the winner. Before the free skate began, all the judges > knew the rank order of the skaters from the technical program and the > importance of the free skate performance in determining the gold > medal winner. This may be why some judging bias was more prevalent > in the free skate than in the technical program." Or, it could be (as the name suggests) that 'free skate' offers more individual choices, more choices that will please or offend personal tastes. > Looney's investigation of the effect of judge's ratings on > the final placement of skaters objectively validates what a chorus of > disbelieving armchair judges had suspected. The median rank system Hooey. You can't 'objectively' validate one set of value-judgments. You can't show that one set of scores arises 'by merit' while another, with exactly the same salient features, does not. The NY Times published the rankings in the pairs free program, the one that ended up with ratings of the French judge being dropped. There were 9 judges, labeled by nationality, and 20 teams. I don't know how the teams were 'qualified' to appear here: there were 3 each, from Canada, Russia, and China. In some sense, anyway, these are the best in the world. I have reproduced the data, below. What astounds me is the uniformity of the rankings. The *worst* Pearson correlation between two judges (also, Spearman, since the scores are ranks) is 0.973, between judges from Japan and Russia. Correlations with the total were above 0.98. The NY Times highlighted the 'discrepancies' between each judge and the Final ranking. Of those 180 rankings, there were two that were off by 3 (Japan rating the U.S. #13 as 10, for instance), 5 that were off by 2, and only 58 others off by 1. The most consistent rankings were by the French judge (the scores that were thrown out). Anyway, one consequence of that 'reliability' is that there is relatively great 'statistical power' for looking at blocs of votes, if such exist. I know some other rankings have been less consistent than this; I don't know how (a)typical this level of agreement might be for this skating event, or others. Personally, I now suspect that there is 'collusion' to the extent that judges agree, before the skate-off, about who will be competing for 1-3 (say), 4-7, ..., 16-20. That might be decided on gross technical competence (again, not invidious). Concerns of great or small errors, difficulty, originality: these play a role within these strata. And, biases about tastes in presentations. *= data: entered (for convenience) by judge. * set up for SPSS to read; transpose; list; correlate. Title Skating Pairs, rankings by judge. data list list / rank1 to rank20 judge(20F3.0,1x,A8). begin data 1 2 3 4 6 5 7 9 8 10 12 11 13 14 15 16 18 17 19 20 Russia 1 2 3 5 4 7 6 8 9 10 11 13 12 15 14 16 17 18 19 20 China 2 1 3 5 4 7 6 8 9 12 10 13 11 14 15 16 17 18 19 20 U.S. 1 2 3 4 5 6 7 9 8 10 11 12 13 14 15 16 17 18 19 20 France 1 2 3 4 5 6 7 8 10 9 11 12 14 13 15 16 17 18 19 20 Poland 2 1 3 7 4 5 6 8 10 9 11 12 13 14 15 16 18 17 19 20 Canada 1 2 3 4 5 6 7 8 9 10 11 12 15 14 13 16 17 19 18 20 Ukraine 2 1 3 5 4 6 7 8 9 11 10 12 13 14 15 16 18 17 19 20 Germany 2 1 3 4 5 7 6 8 9 12 11 13 10 15 14 16 17 19 18 20 Japan end data. execute. flipnewnames= judge. formats russia to japan(F2.0). listall. subtitle'Spearman' is the Pearson corr. compute ranked= $casenum. nonpar corr vars= russia to japan ranked
Re: How to test whether f(X,Y)=f(X)f(Y) is true??
On Wed, 20 Feb 2002 22:21:38 -, "Chia C Chong" <[EMAIL PROTECTED]> wrote: [snip, various discussion before] > > I have an example of data of 2 RVs. When I tested the correlation between > them, by simply find the correlation coefficient, it shows that the > correlation coefficient is so small and therefore, I could say that these > two RVs are uncorrelated,or better still, not linearly correlated. Right! > However, > when I plotted the scatter plot of them, it is clearly shown that one of the > varriable does dependent on the other variable in some kind of pattern, is > just that there are not lineraly dependent, hence the almost zero > correlation coeffiicent. So, I am just wonder whether any kind of tests that > I could use to test dependency between 2 varaibles... Construct a test that checks for features. What features? Well, what features characterize your *observed* dependency, in a generalized way? -- you do want a description that would presumably have a chance for describing some future set of data. The null hypothesis is that the joint density is merely the product of the separate densities. For a picture: a greytone backdrop changes just gradually, as you move in any direction. Distinct lines or blotches are 'dependencies' -- whenever they are more distinct than would 'arise by chance.' The best test to detect vague blotches would not be the best to detect sharp spots, and that would be different from detecting lines. As I wrote before , > > > > So there is an infinite variety of tests conceivable. > > So the *useful* test is the one that avoids 'Bonferroni correction," > > because it is the one you perform because > > you have some reason for it. > > -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: How to test whether f(X,Y)=f(X)f(Y) is true??
On Wed, 20 Feb 2002 19:30:19 -, "Chia C Chong" <[EMAIL PROTECTED]> wrote: > > "Vadim and Oxana Marmer" <[EMAIL PROTECTED]> wrote in message > [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > > You can start with checking if they are correlated. It's simpler to do. If > > you find that they are correlated then you have the answer to your > > question. > > If you find that they are uncorrelated and you have a reason to believe > > that they may be not independent anyway then you can look for more > > advanced tests. > > Can you give some examples of more advanced tests that can be used to test > the depedency of data when there these data are uncorrelated?? You can check for an obvious non-linear (say quadratic) fit. WHAT is your 'reason to believe that they may be not independent'? Anything that makes any pattern, at all, is 'dependent.' So there is an infinite variety of tests conceivable. So the *useful* test is the one that avoids 'Bonferroni correction," because it is the one you perform because you have some reason for it. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: SPC control limits
On 20 Feb 2002 08:50:19 -0800, [EMAIL PROTECTED] (MrTequila) wrote: > Hi all, hope this is the right place. > > i was just wondering what you should do when you establish some > control limits but some of the data points you've just used are > outside of the limits you just established? > > should you write them off as bad, leave them or go back and see/fix > the problem? - looks like an early chance to test them. If you go back and there is *not* any problem, then here's a hint that your limits may not be useful after all. I suppose, if you *know* you did badly, you could write yourself a failing grade -- based on guilt-feelings rather than on data -- and change the limits without ever testing whether there is a real problem. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Numerical recipes in statistics ???
On 19 Feb 2002 13:13:08 -0800, [EMAIL PROTECTED] (The Truth) wrote: > Glen Barnett <[EMAIL PROTECTED]> wrote in message >news:<[EMAIL PROTECTED]>... > > The Truth wrote: > > > Are there any "Numerical Recipes" like textbook on statistics and probability ? > > > Just wondering.. > > > > What do you mean, a book with algorithms for statistics and probability > > or a handbook/cookbook list of techniques with some basic explanation? [ ... ] > > I suppose I should have been more clear with my question. What I > essentially require is a textbook which presents algorithms like Monte > Carlo, Principal Component Analysis, Clustering methods, > MANOVA/MANACOVA methods etc. and provides source code (in C , C++ or > Fortran) or pseudocode together with short explanations of the > algorithms. Unreasonable dreams. Maybe your university would have some trace of the original BMD package. I think it was in the public domain, and that it was the source of many routines for SPSS, SAS, and BMDP. But those were simple routines. That should have Principal components, and perhaps some clustering. However: There are dozens of Monte Carlo methods; you might find a book for whatever your particular *field* happens to be. And check the SPSS 'proximities' routines to get a notion of the complexity of a driver-routine for clustering. It has 3 dozen or so criteria for closeness. Clusters can use [some criterion] and apply it to each case as it relates to each other case, or to each previous group. Or it can relate created-groups to other created-groups. Then there is the big choice of whether cases start separate, to be joined; or start out joined, to become separate. Compared to Clustering, MANOVA might have some pretty good routines. But I have not seen much great program organization for MANOVA procedures, once you get beyond 'discriminant function' and the paradigm of multi-regression with multiple-criteria. 'Pseudocode,' however, is almost what it looks like when you start with matrix routines, in SPSS or R or various specialty programs. You do have to speak the language of matrices, so you know when it is, that you want to get an eigenvector; and what to do with it. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: can multicollinearity force a correlation?
On 18 Feb 2002 16:29:27 -0800, [EMAIL PROTECTED] (Wuzzy) wrote: > > You should take note that R^2 is *not* a very good measure > > of 'effect size.' > > Hi Rich, you asked to see my data, - I don't remember doing that - >i've posted the visual at the > following location http://www.accessv.com/~joemende/insulin2.gif note > that the r^2 is low despite the fact that it agrees with common sense: > Insulin levels are shown here to decrease with increasing exercise as > well as with decreasing food intake.. My r^2 is low but i think it > is clear that the above is true.. > > I've included several different views, "rating" is in MET values, i > forgot to multiply against body weight in kg to get KCAL spent per > day.. If the scattergram is meaningful, blue-ish and to the right, then the big, 3-D shaded bar-graph to the left may be thoroughly misleading. I assume that the plane imposed on both represents the regression. Do the dots in the scattergram mean something? I make sense of them; but then the bars seem to be a bad and misleading choice of plotting something-or-other. The bars are an apparent picture of something greater than R^2 of 2% -- so I am sure they don't represent *that* relationship in a proper way. My tentative conclusion is that your 2% effect really is a small one; it should be difficult to discern among likely artifacts; and therefore, it is hardly worth mentioning -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: covariates !!
On Mon, 18 Feb 2002 20:57:36 +0100, "jan plessers" <[EMAIL PROTECTED]> wrote: > Hello, > > I did a likert on 2 groups, a Mann-whitney showed that ther was a > significant difference between the 2 groups. If 'did a Likert' means what I expect, and the scaling was decent enough to be worth commenting on, then the rank-transformation was unneeded and wasteful of information. > The next thing I did were some > univariate statisics between the covariates and I found some significant > differences between the covariates (ex. age sign. differs between the 2 > groups). > My questions are: > Can I test again for a sign. difference between the 2 groups, but with the > covariates controlled ? How to do this? How to do this in SPSS ? If the groups aren't random, and especially when they are non-randomly non-random, then you have to build the argument that covariates don't matter. First, you can stick them in and see if anything changes. If not, then you are okay. You do want to ask the procedure to interpret the covariates 'before' or 'at the same time as' (not, 'after') the main effects. I've always done this in SPSS (6.1 and earlier) with ANOVA vara by grps(1,4) with covar/ -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Newbie question
On 15 Feb 2002 14:38:49 -0800, [EMAIL PROTECTED] (AP) wrote: > Hi all: > > I would appreciate your help in solving this question. > > calculate the standard deviation of a sample where the mean and > standard deviation from the process are provided? > E.g. Process mean = 150; standard deviation = 20. What is the SD for > a sample of 25? The answer suggested is 4.0 Here is a vocabulary distinction. Or error. I don't know if you are repeating the problem wrong, or you are speaking from a tradition that I am not familiar with. As I am familiar with it, statisticians say that "the standard deviation" is the "standard deviation of the sample." We say that the "standard deviation of the sample *mean*" will be frequently referred to as the "standard error"; and "The SD of the mean [or the SE] equals SD/sqrt(N)". That is confusing enough. I hope this makes your sources clear. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: can multicollinearity force a correlation?
[ snip, previous problem] > > This is similar to a problem I have come across: the measurement of a > serum value against exposure. > My theory is that they are correlated. But the data says that they > have an R^2 of 0.02 even though the p-value for the beta is p=1E-40 > (ie. zero). > > As you explain this is possible. My reasoning is that the exposure is > happening many hours before the measurement of serum and so that is You should take note that R^2 is *not* a very good measure of 'effect size.' It only works when you are repeating something familiar. You have seen so-much before, and you may be happy to see as much again; but it does not tell you as much as knowing that there is a 4-fold Odds Ratio for a factor -- which is the usual measure when you have a rare dichotomy, or something that can be conveniently described that way. [ snip ] > . R^2 is very useful though, for example if > you want to know in the american population what is the highest source > of fat, you would use R^2 on the food frequencies, not the beta > coefficient.. because the R^2 would tell you the food that most > predicts, rather than the "strength" of the effect of the food.. ie. > low fat foods may be main source of fat in diet.. > > -just thinking outloud hehe.. Well, maybe R^2 is useful. But you need to know how it is anchored. Do you have continuous variables? - I thought you had dichotomies, where the Odds Ratio is rule, when you have small rates. A 'coefficient of determination' or R-squared of 0.18 reflects *at least* a 4-fold increase in Odds Ratio when the 4 cells are all around 50% -- For that 0.18, the OR is higher, if the margins are less balanced. And also. The R-squared is going to describe the sample-on-hand: If you sample with too-narrow variation, you get R-squared that is too-small. Similarly, for large. The beta describes co-variation in another way; the raw beta (not the standardized) is usually what is more interesting, if you really have a large enough N that the actual coefficients are interesting (mine usually are not). -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: homogeneity of variance
On 14 Feb 2002 17:14:24 -0800, [EMAIL PROTECTED] (Thomas Souers) wrote: TS> > I have another question regarding one-way ANOVA. I have noticed > that in some books, nothing is said about what you can do when the > factor level variances are unequal. In Neter's big book, > transformations are recommended. " Well, there are a lot of excuses for comparing *two* means. It is tougher to make up reasons why you ought to compare more than two, when they are arbitrary sets of numbers that are not well-behaved. For instance, they are *potentially* well behaved, in my book, if they simply happen to have been recorded or measured in the wrong units/ scale -- so they will be fixed by transformation. You probably ought to be re-thinking your whole scientific hypothesis and test, if your problem is worse than that. TS> > If the data are approximately normal, why not just use a > Satterthwaite approximate t-statistic for pairwise comparisons? > For example, you could use a Bonferroni type procedure. > What are people's opinions about this approach?" You can see my comments (today) on another thread, about t-tests. If Ns are equal, it doesn't matter which test. So you might as well use the regular apparatus. If they are not, both tests are fairly rotten by one-tailed criteria. By the way: What clinical research uses in followup testing for unequal N is either Bonferroni correction, or one or another *approximation* (which may not be very good). (I think I would never trust 'exact procedures' that may have been designed for unequal N, generically.) Personally, I try to do important tests and avoid Bonferroni. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: If T-Test can not be applied
On Thu, 14 Feb 2002 23:48:02 +0100, "Matthias" <[EMAIL PROTECTED]> wrote: > Hello, > > would be nice if someone can give me some advice with regard to the > following problem: > > I would like to compare the means of two independent numerical sets of data > whether they are significantly different from each other or not. One of the > two underlying assumption to calculate the T-Test is not given (Variances > are assumed to be NOT equally distributed; but data is normally > distributed). What kind of (non?)parametric-test does exist - instead of the > T-Test - to calculate possible differences in the two means? [ ... ] The *logical* or measurement problem raised by 'different variances' is manifested when one sample predominates at both extremes. In that case, a conclusion about 'superiority' depends on someone's scaling or weighting of scores. That problem does not exist when the difference between groups is a shift of the mean, or a stretching of the distribution to one side. Comparing *means* ... Are you sure that you want to focus on means? Then it might as well be the t-test, evaluated one way or another: equal variances-assumption; unequal; or randomization. Other than means: The problems that lead to 'other tests' are ones that say the means are 'meaningless', and that you want to look at some version of stochastic superiority. But if the shapes are similar (same family), then the t-test with equal Ns is accurate; and with vastly unequal Ns, the two standard tests are both biased and potentially misleading -- so you have to look at both, and decide whether you should prefer one over the other based on (for instance) where and how the data were collected, or how they arise. If the shapes of distributions are not similar, then rank-transformation is not reliable or robust for 'superiority.' You can compute some Robust tests, such as the 2x2 contingency test, which may be based on median split or whatever split meets your needs. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: one-way ANOVA question
On 13 Feb 2002 09:48:41 -0800, [EMAIL PROTECTED] (Dennis Roberts) wrote: > At 09:21 AM 2/13/02 -0600, Mike Granaas wrote: > >On Fri, 8 Feb 2002, Thomas Souers wrote: > > > > > > 2) Secondly, are contrasts used primarily as planned comparisons? If > > so, why? > > > > > > >I would second those who've already indicated that planned comparisons are > >superior in answering theoretical questions and add a couple of comments: > > another way to think about this issue is: what IF we never had ... nor will > in the future ... the overall omnibus F test? > > would this help us or hurt us in the exploration of the > experimental/research questions of primary interest? - not having it available, even abstractly, would HURT, because we would be without that reminder of 'too many hypotheses'. In practice, I *do* consider the number of tests. Just about always. Now, I am not arguing that the particular form of having an ANOVA omnibus-test is essential. Bonferroni correction can do a lot of the same. It just won't always be as efficient. > i really don't see ANY case that it would hurt us ... > and, i can't really think of cases where doing the overall F test helps us ... > But, Dennis, I thought you told us before, you don't appreciate "hypothesis testing" ... I thought you could not think of cases where doing *any* F-test helps us. [ ... ] -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Book/textbook on applied math with expectation/variance
On Sat, 9 Feb 2002 01:17:14 +0900, "maximus" <[EMAIL PROTECTED]> wrote: > It may seem odd with the question in the title, but I want to read and have > some more > practice with (applied) math with expectation/variance, which is in many > forms, for example > with max/min, integration (inside or outside the integrant, or on the > limit), exponent, etc... > > Not many books offer more than just the simple definitions and examples. > > Is there any particular book that have more ? Level of difficulty? Kendall and Stuart's volumes of "The Advanced Theory of Statistics" could occupy you for a while. Or look in the same section of the library as K&S; or whatever sources you have liked before. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Ansari-Bradley dispersion test.
On Sat, 09 Feb 2002 16:59:34 GMT, Johannes Fichtinger <[EMAIL PROTECTED]> wrote: > Dear NG! > I have been searching for a description of the Ansari-Bradley dispersion test up to >now for > analysing a psychological research. I am searching for a description of this test, >specially a > description how to use the test. > > Please, can you tell me, how to use the test, or show me a link, where it is >described? > Thank you very much in advance, I plugged Ansari-Bradley into a search by www.google.com and there were 287 hits. The first page contained the (aptly named) http://franz.stat.wisc.edu/~rossini/courses/intro-nonpar/text/Specifications_for_the_Ansari_Bradley_Test.html I suggest repeating the search. That also eliminates the "pasting" problem if your reader has broken the long URL into two lines. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: can multicollinearity force a correlation?
On 5 Feb 2002 18:01:15 -0800, [EMAIL PROTECTED] (Wuzzy) wrote: > > You made a model with the "exact same exposure in different units", > > which is something that no one would do, > > Hehe, translation is don't post messages until you've thought them > through. > > Anyway, turns out that the answer to my question is "No".. Well, I think I speak for several statisticians when I say that we still don't know what you refer to as 'multi collinearity'. Do you mean 100%, as in your question? What *are* you asking? > Multicollinearity cannot force a correlation. It turns out that ONE > of the variables *was* correlated With R^2=0.45 and so > multicollinearity had no effect on overall R^2. [ ... ] If you are concluding that you won't improve R^2 by using exactly the same variable twice, you are correct. Another post-er has suggested where you *do* have to watch out for multi-colliinearity: He described the case where the multiple R^2 is large despite small univariate correlations with the criterion. (It was wordier than that, but that is what he did.) For further information-> You could search the last few months of posts in sci.stat.* usinggroups.google.com and look for 'confounding' or 'masking'; and there might be something more in my own stats-faq. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: EViews 3.1 error message
On 6 Feb 2002 11:31:10 -0800, [EMAIL PROTECTED] (James McQueen) wrote: > Hi, > > I tried to run a workfile on EViews 3.1 that I had created in EViews > 4.0. It is a three equation system estimated via full information > maximum likelihood. It worked fine in 4.0. Now, I'm using 3.1 and it > opens the 4.0 workfile no problem but when I hit the estimate button > (via FIML) it returns: error, near singular matrix. As far as I know > the workfile was unchanged from the one that worked fine in 4.0. Do I > need to create a new workfile in 3.1 and re-enter the equations and > re-import the excel file to make it work in 3.1? You are trying to import the 4.0 version of something into its 3.1 version? that sounds like a formula for screwing up, if that package isn't smart enough to prohibit it, or to tell you the explicit limitations or methods. Have you looked for "Export to old version"? Sometimes packages can't read their own *old* versions; and it sounds like you tried to read the *newer* version with an older one. You can try re-importing just the equations (or just the data) separately, but if you have doubts, it might take more time to check the total accuracy than it would to start from scratch. Can you re-import your data from Excel? That sounds safe, even if you create it in Excel by exporting from 4.0. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Case-cross over
On Thu, 7 Feb 2002 13:20:44 +0100, "Anna Axmon" <[EMAIL PROTECTED]> wrote: > Hi, > > does anyone know if there is a textbook on case-cross over design? > "case crossover design" (in quotes) gets 258 hits reported in google. I did not notice a textbook review, but those should lead you to whatever is available. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: time series analysis
- I don't see a public answer to this one from a couple of weeks ago - On 25 Jan 2002 10:16:45 -0800, [EMAIL PROTECTED] (Jack Eslick) wrote: > I am working with a data that were collected at non-uniform time > intervals. I want to use regression analyses to determine if there is > a temporal trend. Common sense tells me that the data need to be > assigned some type of weight value (data collected at closer intervals > should have less ability to influence the regression line). I have > been unable to find a reference about how to assign the weights, other > than some vague references to "an appropriate" method. Can someone > recommend an approachable reference? On careful re-reading, this is not the question that I expected from its first line. Now I gather that the precise lack of uniformity does not bother you for its lack of precision. You are (possibly) quite satisfied with what you have as a time line, for whatever it says about the passage of time. But you have some individuals with many extra (or fewer) points than others, or some parts of the time lines, by individual or for the whole sample, that are not equivalent in data density. - That is fuzzy on my part, but I am not highly confident in (for instance) my basic reading, that you have 'repeated measures.' Why does common sense tell you that you need weights? How extreme is the problem? - Perhaps you are underestimating the robustness of regression. What is the design (N and repeats) and what is the approximate R-squared: Are you trying to model these data on hand with great confidence, and place accurate intervals on coefficients, or are you trying to squeeze out a statistical test that will be legitimate? There is not always a unified solution. For instance, I can imagine that you might be best served by partitioning cases (I am assuming repeated measures again) into 'sparse data' and 'heavy data'; analyzing separately; and combining results. You do have to explain, at some time, why some data are sparse, and argue that it makes no important difference, right? I don't know of particular references for what I have been talking about, but "unbalanced data" might help your search. And you could try us with more detail, if you wish. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Sensitivity Analysis
On 31 Jan 2002 10:06:36 -0800, [EMAIL PROTECTED] (Christopher J. Mecklin) wrote: > I had a colleague (a biologist) ask me about sensitivity analysis. I am > not familiar with the technique (above and beyond knowing that the > technique exists). What books/articles/websites/etc. would be good sources > for my colleague to learn about sensitivity analysis. Since he's a > biologist and not a statistician, I'm assuming he would prefer a treatment > geared towards application rather than theory. I have not seen any reply to this. I suspect that there might be too many options that refer to 'sensitivity' and none of us are sure what you are interested in, precisely. What's another keyword? I pair specificity with sensitivity; but I don't refer to 'sensitivity analysis', I say 'discriminability.' Your question -- and my background thoughts of 1000-generation, simulation analyses in genetic model ling -- makes me think of something I saw years ago, called 'perturbation analyses'. Try Google, or try us again with additional detail. Hope this helps. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Interpreting mutliple regression Beta is only way?
On 4 Feb 2002 16:14:11 -0800, [EMAIL PROTECTED] (Wuzzy) wrote: > > > > In biostatistical studies, either version of beta is pretty worthless. > > Generally speaking. > > If I may be permitted to infer a reason: > if you have > > bodyweight= -a(drug) - b(exercise) + food > > Then the standardized coefficients will affect bodyweight but they > will also affect each other. They would only be useful if drug intake > was perfectly independant of exercise and food in the population. Okay, that is a bit of a reason, which applies widely; but that is not what I was pointing at when I wrote that line. I was over-stating. But many of the results in clinical research are 'barely statistically significant' and when that is so, then (in my opinion), the 95% Confidence Interval does not add much to the statement of "test result", and the point estimate of the effect (beta, or a mean-difference) is pretty loose, too. I want to beat a 5% alpha. And I want a result to have a 50% interval (say) that is actually interesting, and well-above the measurement jitters. -- That is an *essential* requirement for being coherent, if you want to describe observational studies. And it is a pretty good idea for writing about randomized-controlled studies, too. > > If they are not independant but partially collinear (0.5) using linear > regression is it possible to know whether the drug is strong enough > (colloquially speaking) to recommend? > I assume that it would be impossible as a change in drug cannot be > separated from a change in exercise in the population. Ie. people are > exercising and taking the drug so it is impossible to distinguish > which one is beneficial. You can look at the zero-order effect (raw correlation) as well as the partial effect after controlling for other potential predictors: one at a time, or in sets. If your Drug always shows the same prediction, regardless of how you test it, that is a pretty good sign. > I've heard of "ridge regression" will try to investigate this area > more.. > will probably figure it out with time hehe.. Ridge regression can be decomposed into a combination of the p-variate regression, averaged with the 1-variable regressions, and all the i-variable regressions that lie in-between. There is not much gain using Ridge if you avoid suppressor relationships from the start -- all those cases where a beta is the opposite sign from its zero-order beta. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: can multicollinearity force a correlation?
On 5 Feb 2002 08:28:05 -0800, [EMAIL PROTECTED] (Wuzzy) wrote: > Is it possible that multicollinearity can force a correlation that > does not exist? > > I have a very large sample of n=5,000 > and have found that > > disease= exposure + exposure + exposure + exposure R^2=0.45 > > where all 4 exposures are the exact same exposure in different units > like ug/dL or mg/dL or molar units. Now, see, that is totally, thoroughly ignorant. You made a model with the "exact same exposure in different units", which is something that no one would do, who had understood even one single semester of intro-to-statistics. > > Nonetheless when I do a simple correlation (pearson) I found that the > exposure in ug/dL did not affect the disease. > I find it extremely likely that you do not know how to read the computer printout and then tell us what you have read. > This seems hard to believe as my sample is relatively large.. > I don't believe the 0.45 R^2 is possible but was shocked by it. I'll > try to rerun it in other, more realistic models. My advice is: Please ask for local help from someone who can lead you through analyses, step by step. If you insist on asking like this, I suggest that you cut-and-paste some computer output; or e-mail the full dataset to your eventual e-mail helper. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: The exam questlons will probably be tougher
On 3 Feb 2002 03:34:35 -0800, [EMAIL PROTECTED] wrote: [ snip, previous examples ... ] > > Asked about statistics practice problems. Search took 0.35 seconds > Got about 945,000. hits. > > Nuff said, - Right idea, sloppy technique; I would say in my critique. Google hits 966,000 for me, for , but that means it found more than one of the three words, somewhere. Probably. Too many to scan much of. When you get that sort of excess, you should choose to guide the search a little more, by using Advanced Search (or using quotes to get the same thing). There were only 67 for <"statistics practice problems">, some of them designed for a particular class's test. Quick inspection shows that the latter two words should be promising. nets 7420, so you might be able to select topics within those. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: R-square & parameter estimates
On 31 Jan 2002 08:26:19 -0800, [EMAIL PROTECTED] (Yvette) wrote: > Hi all, > > My question, although probably basic to most of you, is: If you are > comparing two models, why might the test variables parameter estimates > be significant in the second case and not in the first yet the > R-square is decreased. For example: > [ snip, rest ] "Comparing two models" usually means something in particular to statisticians: there is an overall test. That is not what you mention or describe. I can construct for you a HIGHLY accurate model with several variables, where none of the individual coefficients (which measure 'partial' effects) will be important. - look up subjects like, "interpreting (partial) coefficients" and discover how awkward that is. Hope this helps. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Verba Volant 31-01-02
On Thu, 31 Jan 2002 10:37:13 -0500, Rich Ulrich <[EMAIL PROTECTED]> wrote: > Postmaster, > Would you please tell your User that they have to > stop sending their daily message to Usenet Newsgroups. > > Below are the headers, and the contents of what was > received today, at sci.stat.edu. > > Rich Ulrich, [EMAIL PROTECTED] > == headers and message (raw plus HTML plus encoded) ooops, sorry folks, my apologies. I think I sent that to the proper postmaster; but I thought I was *not* sending it to the newsgroup. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Verba Volant 31-01-02
Postmaster, Would you please tell your User that they have to stop sending their daily message to Usenet Newsgroups. Below are the headers, and the contents of what was received today, at sci.stat.edu. Rich Ulrich, [EMAIL PROTECTED] == headers and message (raw plus HTML plus encoded) Path: pitt.edu!nntp.club.cc.cmu.edu!micro-heart-of-gold.mit.edu!enews.sgi.com!howland.erols.net!newsfeed.lotsanews.com.MISMATCH!pln-e!spln!dex!newsgate.newsguy.com!newsp.newsguy.com!mdrn From: [EMAIL PROTECTED] (Verba Volant) Newsgroups: sci.stat.edu Subject: Verba Volant 31-01-02 Date: 31 Jan 2002 03:56:55 -0800 Organization: None Lines: 293 Message-ID: <5.1.0.14.2.2002013927.00a4a210@localhost> NNTP-Posting-Host: mail2.rol.it Mime-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="=_12409671==_.REL" NNTP-Posting-Date: 31 Jan 2002 11:56:55 GMT Xref: pitt.edu sci.stat.edu:32797 --=_12409671==_.REL Content-Type: multipart/alternative; boundary="=_12409687==_.ALT" --=_12409687==_.ALT Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Verba Volant 31-01-02, Every day a new quotation translated into many languages. <http://www.logos.net/bimbi/index_en.html>childrens_dictionary.png _ Quotation of the day: Author -=20 <http://www.google.it/search?q=3DElisa+F%E9lix&hl=3Dit&btnG=3DCerca+con+Goog= le&lr=3D>Elisa=20 <http://www.google.it/search?q=3DElisa+F%E9lix&hl=3Dit&btnG=3DCerca+con+Goog= le&lr=3D>F=E9lix English - only that which I have lost belongs to me for always Italian - soltanto ci=F2 che ho perso mi appartiene per sempre Spanish - tan s=F3lo lo que he perdido me pertenece para siempre French - seul ce que j'ai perdu m'appartient =E0 jamais German - nur dass, was ich verloren habe, geh=F6rt mir ewig Albanian - vet=EBm =E7'kam humbur m=EB p=EBrket p=EBrgjithnj=EB Basque - galdu dudana bakarrik dut betiko nire Bolognese - s=E5ul qu=E0ll ch=92ai =F2 p=EArs al m apart=E9n par s=E4nper Bresciano - s=F9l ch=E8l che go pird=ECt l=92=E8 l=92m=E9 per s=E8mper Calabrese - sulu chiru ca 'gi=F9 persu =E8 miu pi sempri Catalan - nom=E9s all=F2 que he perdut, ser=E0 meu per sempre Croatian - samo ono =9Ato sam izgubila je moje zauvijek Czech - jen to, o co jsem pri=9Ala, mi patr=ED nav=9Edy Danish - kun det jeg har misted tilh=F8rer mig Dutch - alleen wat ik heb verloren, behoort me voor altijd toe Emiliano Romagnolo - sulment quel c'a io smarid l'=E8 par semper cumpagn ma= me Esperanto - nur tio kion mi estas perdinta apartenas al mi ciam Ferrarese - sol qu=E9l ca i'o pers m'aparti=E9n par sempar Finnish - vain se mink=E4 olen menett=E4nyt on ikuisesti minun Flemish - alleen wat ik heb verloren, behoort me voor altijd toe Galician - s=F3 aquilo que perd=EDn me pertence para sempre Hungarian - csak az tartozik hozz=E1m mind=F6r=F6kre, amit m=E1r= elvesz=EDtettem Latin - tantum quod perdidi meum in perpetuum est Latvian; Lettish - tikai tas, ko esmu zaudejusi, man pieder uz mu=9Eiem Leonese - nam=E1i aqueillu que perd=ED pertenezme pa siempres Mantuan - sol quel ch'=F2 pers 'l =E8 mio par s=E9npar Mapunzugun - =F1i fij elk=FCnuel m=FCten inchegefuyem Neapolitan - sulamento chello ca aggiu perduto =E8 pe ssempe d'o mio Occitan - masque =E7=F2 qu=92ai perdut es miu per sempre Parmigiano - soltant col che g'ho pers l=B4=E9 m=E9 per semper Piemontese - mach l=F2n ch'i l'hai perd=F9 a l'=E9 m=E8 p=EBr sempe Polish - tylko to co stracilem na zawsze mi przynalezy Portuguese - s=F2mente isso que perdi sempre pertenecer=E1 a mim Reggiano - sol c=F2ll ch'j=F2 peers l'=E9 mio per s=E8imper Roman - solo quello che ho smarito =E8 mmio pe' sempre Romanian - doar ceea ce am pierdut, =EEmi apartine pentru totdeauna Sardinian (Limba Sarda Unificada) - petzi su chi apo p=E8rdidu m'at a=20 apartenner pro semper Sicilian - sulu chiddu c' haiu pirdutu...m'apparteni 'ppi sempri Slovak - iba to, co som stratil, mi patr=ED nav=9Edy Swedish - bara det jag f=F6rlorat =E4r mitt f=F6r all framtid Turkish - sadece kaybettigim, her zaman bana aittir Venetian - solo quel che go perso el sar=E0 mio par senpre Zeneize - s=F6o che quello ch'=F2 perso o l'=E9 o m=E6 pe de longo _ All languages, please click on this link http://www.logos.net/owa-l/press.frasiproc.carica?code=3D508 _ To unsubscribe from Verba Volant, please follow this link: http://www.logos.net/owa-l/press.rol_ml.verbavolant1?lang=3Den and write in the empty field next to unsubscribe the email address that you= =20 find after "TO:" in the Verba Volant emails alternatively write to the following address: [EMAIL PROTECTED]= =20 always co
Re: area under the curve
On 30 Jan 2002 08:02:51 -0800, [EMAIL PROTECTED] (Melady Preece) wrote: > A student wants to know how one can calculate the area under the >curve for skewed distributions. Can someone give me an answer about >when a distribution is too skewed to use the z table? It would be convenient if there was a general answer. Here are some reasons why there isn't one answer: - Sometimes the accuracy of F(z)-for-F(x) is needed within a certain absolute amount (like the accuracy for a KS goodness of fit). Skewness does not index this. - Often, the accuracy of F(z) is desired within 'so-many decimals' of accuracy: two or three or more places of accuracy for both F(z) and 1-F(z). Skewness does not index this. In short: there are several ways to ask for accuracy, but there is no automatic (or pre-computed) connection that I have heard of, between skewness and the accuracy of F, for any measure of accuracy. I think I would find it interesting to see some Monte Carlo experiments. I think those should start with specific classes of distributions, and specific ranges for their parameters, and plot their 'accuracies' against the observed skewness and kurtosis. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: cutting tails of samples
On 17 Jan 2002 00:05:02 -0800, [EMAIL PROTECTED] (Håkon) wrote: > I have noticed a practice among some people dealing with enterprise > data to cut the left and right tails off their samples (including > census data) in both dependent and independent variables. The reason > is that outliers tend to be extreme. The effects can be stunning. How > is this practice to be understood statistically - as some form of > truncation? References that deal formally with such a practice? This is called "trimming" - 5% trimming, 25% trimming. The median is what is left when you have done "50% trimming." Trimming by 5% or 10% reportedly works well for your measures of 'central tendency', so long as you *know* that the extremes are not important. I don't know what it is that you refer to as 'enterprise data.' -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: factor Analysis
On Tue, 29 Jan 2002 10:52:30 +0100, "Huxley" <[EMAIL PROTECTED]> wrote: > > Uzytkownik "Gottfried Helms" <[EMAIL PROTECTED]> napisal w wiadomosci > [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > > It's not so simple. You have to do matrix-inversion for > > that. > > > Not simple? I heard that taking suitable factor loadings and every variable > mean I can obtain this space. e.g. (I do not know is it true) > Let mean for car1 and questions 10 (variables): > mean X1=1 > mean X2=2 > .. > mean X10=10 > I have 2 factor score. > factor loadins (aij) I have, therefore for first factor score, co-odrinate > for car1 is > F1(for car1)=1*a(1,1)+2*a(2,1)+3*a(3,1)+...+10*a(10,1) > is it true? No, that is not true. Please believe them. Factor loadings are *correlations* and serve as descriptors. They were neither scaled nor computed as regression coefficients - which is what you are trying to use them as. Now, in clinical research, we don't usually bother to create the actual, real, true factor, for our practical purposes. For practical purposes, it is important to have some face-validity for what the factor means. And it is handy for replication, as well as for understanding, if we construct a factor as the summed score (or average score) of a set of the items. So I look at the high loadings. For a good set of items, it can be realistic and appropriate to 'assign' each item to the factor where its loading is greatest, thus using each item just once in the overall set of several derived factors. (For a set of items where many items were new and untested, it can be appropriate to discard some of items -- where the loadings were split, or were always small.) Each factor is scored as the average score for of a subset of items. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Interpreting mutliple regression Beta is only way?
On 18 Jan 2002 16:55:11 -0800, [EMAIL PROTECTED] (Wuzzy) wrote: > Rich Ulrich <[EMAIL PROTECTED]> wrote in message > > Thanks Rich, most informative, I am trying to determine a method of > comparing apples to oranges - it seems an improtant thing to try to > do, perhaps it is impossible . Well, I do think there is a narrow, numerical answer about two particular tests, in a particular sample. But there are a lot of side-issues before you get to an answer that is very useful. See below. > > I am trying to > determine which is better, glycemic index or carbohydrate total in > predicting glycemic load (Glycemic load=glycemic index*carbohydrate). > > my results as a matrix: > > GI load GI Carb > GI load 1.000 > GI .5331.000 > Carb .858.1241.000 > > So it seems that carb affects GI load more than does GI.. but this is > on ALL foods.. (nobody eats ALL foods so cannot extrapolate to human > diet) but I don't think you're allowed to do this kind of comparison > as Carb and GI aretotal different values: > > I suspected that you would be allowed to make the comparisons if you > use Betas, ie. measure how many standard deviation > changes of GI and Carb it requires.. If it takes a bigger standard > deviation of Carb then you could say that it is more likely that carb > has a bigger effect on glycemic load. > > you seem to suggest that even using standard deviation changes, you > cannot compare apples to oranges. Which sounds right but is > dissapointing.. There is a narrow, numerical thing with an answer. For instance, if you are adding A+B=C, then two *independent* components of C "affect" C in proportion to their variances. Your two components don't have much correlation (.13) -- that is, they are nearly independent -- so that would work out. But you actually have an *exact* relationship, as a product. The one that "matters" in this case is the one that has the higher "coefficient of variation" -- the larger standard deviation (and variance) when expressed as logs. In a sum of two numbers, the one that is "more varying" will "contribute more." You can state that A or B is bigger, for *this* particular sample. Now, I imagine a sample could be a) all healthy; b) all with a 'definite' diagnosis ; c) all in the category of needing a diagnosis. Or you could have some mixture of the above; for some specified ages, etc. Now, What is it you are trying to decide? - that will help determine a relevant sampling. When you say that a number is "more important", are you trying to say that the measure you have says a lot? - or that *improving* that particular number would gain you more, because the number you have is a lousy one? -- I can point out that if the two variables matter exactly the same in some physiological sense, then you can have two opposite conclusions about which is "important," if one of them is *measured* much more poorly (much more inherent error; measurement error) than the other. And, you are reporting on tests, and a number. You don't get to conclude that "glucose matters more than carbohydrates" if you only know that your 50 mg glucose test is more pertinent than your 50mg c. test; as collected in your particular sample. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: coefficient in logistic regression
On 25 Jan 2002 08:13:41 -0800, [EMAIL PROTECTED] (Claudiu D. Tufis) wrote: > Hi, > > I have a multiple logistic regression. Among the predictors, I have 6 > variables that represent the dummies for an interaction term (the > seventh is the reference category and is not included in analysis). I > have obtained for five of these variables extremely large coefficients: > exp(b) ranges from 90,000 to 166,000. > > Could you please tell me if it is normal to have such values for exp(b)? > Do you think it is something wrong? > No, it is not normal. Yes, something is wrong, if that is really what you have. If you have more question, copy some output for us; send some set-up lines; and mention what program. Also, you posted many, many lines of HTML; please turn off that option if you can figure out how. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: QUERY on multiple linear regression: predicted values show much smaller ranges
On 24 Jan 2002 07:09:23 -0800, [EMAIL PROTECTED] (Rich Einsporn) wrote: > Jim Clark gave a fine answer to the question posed by Sangdon Lee. > However, I am curious about the correlation and R-square figures given by > Sangdon. Apparently, the R-squares for the simple linear regressions on > X1 and X2 are (-.2)^2 = .04 and (.3)^2 = .09, but Sangdon says that the > R-sq for the multiple regression is "ONLY" 0.3. I find this to be > surprisingly high, not low. In the examples I see, the R-sq for the > combined model is at most the sum of the individual R-squares. Is it even > possible for the opposite to occur? "Is it possible?" Certainly. The predictors have to be correlated for that to happen; and these were, at 0.6. (Plug all the r's into the equation for multiple-R, and you can check his total. I did not check because it looked feasible, to my eyeball.) "Confounding". It is more common for two predictors to be highly correlated, and share their prediction variance, so that the total R^2 is barely more than either one alone. But these two variables, correlated 0.6 (which is pretty high), predict in opposite directions; so their joint prediction will be greater than the sum. (This relates to another recent question: At the extreme, this negative confounding is what can produce standardized betas that are greater than 1.0. My main use of standardized betas is to compare them to the zero- level r's in order to judge confounding, in either direction. I don't know of a regression statistic that directly tells me that; maybe I should create one.) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: analyzing counts from ordered categories
On 23 Jan 2002 07:52:49 -0800, [EMAIL PROTECTED] (Mike Granaas) wrote: [ snip ] > > ... Their > goal was then to test whether the pattern of endorsements was indeed > > # item 1 > # item 2 > # item 3 > # item 4 > # item 5 > > where "# item 1" is short for "number of individuals endorsing item 1", > and so on. > > I know how to test for equality of proportions. I even know how to test > for specific hypothesized proportions. But in this case there were no > specified proportions, only a hypothesized ordering of proportions, I have > no idea what to do. > > I could make up a set of proportions that fit with the hypothesis, but > there are many such sets and so this seems hardly satisfactory. > > Any ideas? > > I am also leary of counting folks multiple times. In this case if someone > endorsed items 1, 2, and 3 they were counted in all three categories > rather than, say, the highest category endorsed. I have a vague > recollection of some means of dealing with repeated measures count data, > but certainly not recall what it was or under what circumstances it might > be appropriate. For endorsements of #1 and #2, you have - people who endorsed neither; - people who endorsed both; and - people who endorsed one or the other. If 1 & 2 are not ordered, 1-not-2 will equal 2-not-1. So you can do McNemar's test on the two categories, to show whether one was endorsed more than the other. Et cetera. (There is also a multiple category test, but I am not sure whether it would fit.) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: how to adjust for variables
On 21 Jan 2002 16:53:31 -0800, [EMAIL PROTECTED] (Wuzzy) wrote: > Pretend you want to see how fat relates to cancer risk > > fat Kcalcancer > 1 2 100 > 2 4 120 > 3 6 130 > 4 8 140 > 5 10 150 > 6 12 160 > 7 14 170 > 8 16 180 > 9 18 190 > 1020 200 > > You have to adjust for KCal, but how is this done, is the following > the BEST way? The problem is *nonsense* as it is stated, since fat= KCal except for the measurement units. Adjusting A for A', where A' is approximately A except for irrelevant measurement error, you have essentially nothing left. Ever. You always have to be careful when you adjust for something that has much correlation, to be sure that the direction of the logic thoroughly makes sense. (If there is not much correlation, then there is not much change possible in the estimator -- though a *test* could become more significant if a bunch of error variance is accounted for.) A simple way to "adjust for sex" (for instance), is to compute a statistic for each sex separately, and then average. "Within group" is the basic idea of adjusting. [ ... ] > Is doing a univariate regression between the variable you want to > adjust for and your predictor the only way to adjust for values as Univariate? Absolutely not. *Multiple* regression gives "partial regression coefficients." Those "adjust." > above? Studies often cite how they have "adjusted" for KCal, is this > the way they do it, they usually do not specify the method. The method is there, ordinarily, in the articles I read. It is a poor journal (I think..., I estimate blindly ...) that does not require a statement of that method, but if you know nothing of regression, etc., you won't recognize the statement when it appears. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: 95% CI for a sum
On 21 Jan 2002 08:55:21 -0800, [EMAIL PROTECTED] (Scheltema, Karen) wrote: > I have 2 independent samples and the standard errors and n's associated with > each of them. If a and b are constants, what is the formula for the 95% > confidence interval for > (a(Xbar1)+b(xbar2))? > Var(A+B) = Var(A) + Var(B)for A,B independent; and Var(kA) = k^2 Var(A) Right? The SE is the SD of the mean, so you have the terms you need. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Likelihood ratios and non-nested models
On 19 Jan 2002 10:25:10 -0800, [EMAIL PROTECTED] (Brian Leung) wrote: > Greetings, > > I've read that the likelihood ratio test is not valid for non-nested > models. Is this still true if the PF (i.e., multinomial) is the same, > but the link function differs. > Yes, it is still true. It is the subtraction that allows the estimates to be *independent* so that the result may be readily interpreted. However, it is also true that people decide and publish using related criteria for non-nested models. Such conclusions are not as rigorous, but seem to work in various applications. Your comparison of link functions sounds familiar, for instance. Here is one link I found by searching on < BIC "information criterion" > , http://www.saam.com/faq/saam2/right.html (I keep forgetting how to spell AKAIKE.) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Interpreting mutliple regression Beta is only way?
On 16 Jan 2002 11:33:15 -0800, [EMAIL PROTECTED] (Wuzzy) wrote: > If your beta coefficients are on different scales: like > you want to know whether temperature or pressure are affecting > your bread baking more, > > Is the way to do this using Beta coefficients calculated > as Beta=beta*SDx/SDy Something like that ... is called the standardized beta, and every OLS regression program gives them. ... > > It seems like the Beta coefficients are rarely cited in studies > and it seems to me worthless to know beta (small "b") as you are > not allowed to compare them as they are on different scales. In biostatistical studies, either version of beta is pretty worthless. Generally speaking. What you have is prediction that is barely better than chance. The p-values tell you which is "more powerful" within this one equation. The zero-level correlation tells you how they related, alone. -- If these two indicators are not similar, then you have something complicated going on, with confounding taking place, or joint-prediction, and no single number will show it all. - When prediction is enough better-than-chance to be really interesting, then the raw units are probably interesting, too. [ ... ] > Is there a way of converting this standardized coefficient to a > "correlation coefficient" on a scale of -1 to +1) > It would be useful to do this as you want to know the correlation > coefficient of temperature after factoring out pressure. I think you are looking for simple answers that can't exist, even though there *is* a partial-r, and the beta in regression *is* a partial-beta. The main use I have found for the standardized (partial) beta is the simple check against confounding, etc. If ' beta' is similar to the zero-order r, for all variables, then there must be pretty good independence among the predictors, and interpretation doesn't hide any big surprises. If it is half-size, I look for shared prediction. If it is in the wrong direction or far too big (these conditions happen at the same time, for pairs of variables), then gross confounding exists. Hope this helps. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: a problem
On Tue, 15 Jan 2002 23:06:25 GMT, janne <[EMAIL PROTECTED]> wrote: > Lets say I do a x2(chi) test and have the hypothesis: > [snip, some example] > > If you can have < in hypothesis, then when is it < and when is it > I > should use? How do I know which one to use? > > I also wonder about t-tests the same question. When do I know if I > should use < or >? Are you referring to the chisquared test on a contingency table? That is the most popular thing called 'chi-squared test' but it is far from the only thing. Almost always (but not always), chisquared is used to 'reject' when the chisquared value is large. Now, if your H0 and Ha require "less than," that is a pretty good indicator that you should *not* be using the contingency table; but you might be using a 'test for proportions' that gives you the square root of a 1-d.f. chi-squared, which is a normal-deviate z: which has either a plus or minus sign attached. But you *can* use the chisquared test, and make sure the differences are in the right direction. Short answer: Look at the numbers, and use your head. There is not a magical formula that makes a stupid-looking answer come out to be correct. If this still seems confusing, borrow a book or two on experimental design and spend time on the earliest chapters. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Empirical data Fitting
[ rearranging to the usual order, with Reply at the bottom ] > Chia C Chong wrote: > > > > Hi!! > > > > I have a set of data with some kind of distribution. When I plotted the > > histogram density of this set of data, it looks sth like the > > Weibull/Exp/Gamma distribution. I find the parameters that best fit the data > > and then, plot the respective distribution using the estimated parameters on > > the empirical distribution. My question is, what kind of statistical test > > that I should ... [ ... ] On 14 Jan 2002 08:15:35 -0800, [EMAIL PROTECTED] (kjetil halvorsen) wrote: > A quantile-Quantile plot for graphical comparison is best, if you need a > numerical test you can use the pearson correlation coefficient between > the observed and expected quantiles. A table for that test you can ake > for yourself with simulation. No, I think that advice is not good, in this instance. Someone else has mentioned that these three belong to the same family so that it is possible and most desirable to solve for the value of the parameter that distinguishes them. The correlation between those 'quantiles' -- That (I think) gets you a test like the Shapiro-Wilks which is a fine test for normality when you compare correlations across a range of deviant samples. I don't remember its analog being used for testing between theoretical CDFs; and I suspect it is inferior to other approaches for (say) strongly skewed distributions. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Fisher & Tippett Extreme Value Theorem
On 10 Jan 2002 03:28:29 GMT, [EMAIL PROTECTED] (Vmcw) wrote: > Hi! > > I'm looking at an article that references a result by Fisher & Tippett > (specific reference is "Limiting Forms of the Frequency Distribution of the > Largest or Smallest Member of a Sample," Procedings of the Cambridge > Philosophical Society 24, 180-190 (1928)). The article presents the result as > follows: if there are S independent samples of size m from a parent population > bounded below by a, and if xi is the smallest value in sample i, then the > limiting distribution of the minimum of the xi-values is Weibull with location > parameter a as m -> infinity. > > This seems like a not particularly rigorous description of the result, and I > wanted to read the original statement to make sure that all the details are > covered here, i.e. it seems like the parent distribution would have to be > continuous, at least, and that a should be the greatest lower bound rather than > just a bound. So, I ordered the original article from my library and have been The way that I read your 1928 summary, I don't figure out why you are looking for a 'bound.' It says, some distribution becomes (increasingly) Weibull. With a parameter. Where do you see a bound? And, yes, I would guess that somewhere they say something is continuous. > waiting a lng time with no success. So... can anyone help me out with a > precise statement regarding this result? (If it helps, I have obtained > Gumbel's "Statistics of Extremes" to see if it's buried in there, but so far > haven't found it. Page number?) > -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: SAT Question Selection
On Sun, 13 Jan 2002 13:04:14 GMT, "L.C." <[EMAIL PROTECTED]> wrote: > Back in my day (did we have days back then?) I recall > talk of test questions on the SAT. That is, these questions > were not counted; they were being tested for (I presume) > some sort of statistical validity. > > Does anyone have any statistical insight into the SAT question > selection process. Does anyone have a specific lead? I can > find virtually nothing. I believe that they have to change their questions a lot more often than they used to, now that they occasionally reveal some questions and answers. The Educational Testing Service has a web site that looks pretty nice, in my 60-second opinion. http://www.ets.org/research/ They do seem to invite communication -- I suggest you e-mail, if you don't find what you are looking for in their 8 research areas, or elsewhere. It seems to me that I found a statistics journal produced by ETS when I was looking up references for scaling, a year or so ago. But I don't remember that for a fact. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Buy Book on "Probability and statistical inference"
On Sat, 12 Jan 2002 14:37:10 -, "Chia C Chong" <[EMAIL PROTECTED]> wrote: > Hi! > > I wish to get a book in Probability and statistical inference . I wish to > get some advices first..Any good suggestion?? a) Browse in a good college bookstore. There should be a section among general books, in addition to the textbooks that are filed by course. You can not the titles, even if the books are sealed in plastic. Anything that has been in print for decades is good (for something). b) Look for some of those titles in a good college library, and go to the shelves where you would find them. Browse the contents to see what you are looking for. - Then you could ask a stat-group for informed opinions, if you don't get titles in response to this first request. Further, online: you could do a http://groups.google.com search of the stat-groups. You could go to stat-web pages of people who answer questions in the stat-groups, and look for references and links to references. Hope this helps. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Proportionate vs. disproportionate
On 11 Jan 2002 07:46:20 -0800, [EMAIL PROTECTED] (Dennis Roberts) wrote: > definition: proportionate = equal % change > > IF we agree on this ... and maybe we don't ... then, since the % change is > always UN =, then all changes are DISproportionate [ ... ] Are you sure you *advocate* that? I am not positive, but I think I would have objected to "equal % change" as =proportionate= by the time I finished algebra in high school. I know I have objected to similar confusion, on principled grounds, since I learned about Odds Ratios. I suspect that the original sample was small enough that the apparent difference in ORs was not impressive. -- RIch Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: what is the appropriate method?
On Thu, 10 Jan 2002 12:00:06 +0100, "Jos Jansen" <[EMAIL PROTECTED]> wrote: > > "Rich Ulrich" <[EMAIL PROTECTED]> wrote in message > [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > > On Wed, 09 Jan 2002 08:33:41 GMT, [EMAIL PROTECTED] > > (Jukka Sinisalo) wrote: > > > > > > > > We have two pots with 25 plants each. After an identical treatment we > > > wait for a week, and then calculate how many of the plants died. We > > > repeat this "experiment" 20 times, so we end up with 20 pairs of > > > "survival percantages". > > > > > > We are interested in determining the accuracy/reliability of our > > > method. In other words if in the future we use just one pot of 25 > > > plants, what will be the confidence interval of the result. > > snip > > > "transform the data" - is easy and apt. > > > > Compute the logit and use that in your modeling. With > > the difference of two of them, you have the "log Odds Ratio." > > snip > > Using logits is obvious, but log Odds Ratio is not, given the aim to use > only one pot in the future (not the difference of two). An estimate of the > sum of variance components within and between repeats will be required for > calculating the precision of a single result. > Oh, right. Thanks. I was reading the problem as something different. Misreading it. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: what is the appropriate method?
On Wed, 09 Jan 2002 08:33:41 GMT, [EMAIL PROTECTED]
(Jukka Sinisalo) wrote:
>
> We have two pots with 25 plants each. After an identical treatment we
> wait for a week, and then calculate how many of the plants died. We
> repeat this "experiment" 20 times, so we end up with 20 pairs of
> "survival percantages".
>
> We are interested in determining the accuracy/reliability of our
> method. In other words if in the future we use just one pot of 25
> plants, what will be the confidence interval of the result.
>
> At the moment we calculate the pairwise differences,
> and use the standard deviation of those 20 differences to
> estimate the uncertainty of our method. The standard deviation
> I end up is relatively large, and if the survival percentage happens
> to be high, I get a funny confidence interval where the upper limit
> is > 100%, which is clearly impossible.
>
> I have a feeling this is not the best way to go about this,
> but I'm unsure what to do. Perhaps use maximum likelihood
> probabilities for P("plant dies"), or somehow transform the
> data?
"transform the data" - is easy and apt.
Compute the logit and use that in your modeling. With
the difference of two of them, you have the "log Odds Ratio."
The logit is a natural for modeling growth-within-limits;
and it works rather well for modeling percentages.
For your example, you will probably end up talking
about the log of the Odds Ratio for each week.
Among other advantages, it has this one that you mention:
When you back-transform, the CIs are never improper.
The logit is < log( p/(1-p) ) >
You can't compute LOG if the survivorship is ever 0 or 100%.
Folks usually plug in 0.5 for 0 cases when that happens.
(Starting with 25 plants, plug in 2% or 98% for 0 or 100%.)
--
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html
=
Instructions for joining and leaving this list, remarks about the
problem of INAPPROPRIATE MESSAGES, and archives are available at
http://jse.stat.ncsu.edu/
=
Re: Chart of the Week
On Wed, 09 Jan 2002 00:47:19 -0500, Derrick Coetzee <[EMAIL PROTECTED]> wrote: > Gary Klass wrote: > > > http://lilt.ilstu.edu/gmklass/COW/ > > As other posters have noted: always beware the "obvious" implications of > correlations. A common example is that drownings and ice cream sales are > strongly correlated. [...] Yesterday's newspaper summarized a report that screen-writers who had more movies to their credit, and more 4-star pictures in particular, were more likely to have won a writing Oscar, instead of just being nominated. No! Wait! It was backwards of that -- the authors (so it seemed in the news item) concluded that Winners should go on to be more successful... but die younger. And further speculation. My interest faded before I looked it all up. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Is there a problem with this kind of comparison?
On 3 Jan 2002 20:17:48 -0800, [EMAIL PROTECTED] (Lucas Wells) wrote: [ snip, detail ] > > Now, I look at these percentages and I think to myself, 'They're > percentages of a whole. If one goes up, then another must fall. It > doesn't seem to make sense to examine them as if they are measures > that can be seperately influenced (ie, as if we could decrease > percentages across the board).' > > Is this a legitimate concern? > > I could understand it, if one type of error was more 'important' than > another, then perhaps you would be trying to minimise the percentage > of that particular error, but you would expect the others to inflate > as a result, yes? Absolutely right. For the numbers given, the report is silly. (Your example has just one error per-thing. Does the big report look much different with multiple errors?) There could be rates of errors per transaction; per section-of-a-transaction; per total-errors. Whatever exists. The rate or fraction of errors per transaction seems to be a standard that would be worth tracking across reports. The fraction per all-ERRORs seems pretty useless, unless, for instance, you were in a bad state where (say) 90% of all errors were one kind, so you were trying to bring that one sort down. Hope this helps. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: Empirical data Fitting
On Thu, 3 Jan 2002 01:02:17 -, "Chia C Chong" <[EMAIL PROTECTED]> wrote: > Hi Bill..Thanks for your reply. You mentioned in the last line of your > message that statistical tests are not a very good way to choose among > distributions. If this is the case, what test do you think is better in my > case?? ... Other that statistical? - Being familiar; being logically inherent; being brief... Occam's razor goes something like this, "Do not unnecessarily multiply entities in an explanation." -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: One-tailed, two-tailed
On Sun, 30 Dec 2001 18:07:16 -0500, [EMAIL PROTECTED] (Stan Brown) wrote: > Rich Ulrich <[EMAIL PROTECTED]> wrote in sci.stat.edu: [ ... ] RU > > We should > >not overlook the chance to teach our budding statisticians: > >*Always* pay attention to the distinction between random trials > >or careful controls, on the one hand; and grab-samples on the other. > >[Maybe your teacher asked the question that way, in order to > >lead up to that in class?] > SB > [ snip; was in a book of homework problems ...] RU > > >The numbers do not *prove* that one gas gives better mileage; > >the mileage was, indeed, better for one gas than another -- for > >reasons yet to be discussed. Different cars? drivers? routes? SB > > All good points for discussion. But I wouldn't focus too much on > that off-the-cuff word "prove". (I'm not being defensive since I > didn't write the exercise. :-) My students did understand that > nothing is ever proved; that there's still a p-value chance of > getting the sample results you got even if you did perfect random > selection an d the null hypothesis is true. Maybe I'm being UNDER- > scrupulous here, but I think it a pardonable bit of sloppy language. > I read this, and I re-read it, and it still seems like Stan has missed the gist. Type I error ? Stan sees all the problem (so it seems to me) as being one of Type I error, "there's still a p-value chance..." Yes, you *might* construe *some* of the problem that way, some of the time. But I am concerned with the fundamental difference, randomized trials versus grab-sample (observational): For the former, you get to start with the assumption that outside variables are not important. For the latter, you *have to* start with the assumption -- if you are a properly trained scientist -- that they are *apt* to be. It is not enough that you have a 'single, important hypothesis' like you have in a design; you have to explain away the other possible causes. RA Fisher (along with a few others) defended tobacco for years. The case against tobacco was not solidly made until *other*, competing explanations had been laid to rest. That included a convergence with biomedical evidence. I was annoyed by sloppiness when I was in grad-school. If I remember right, it was the epidemiologists who regularly were sloppy, and failed to qualify their tests, while the math-trained statisticians were apt to be clear. "We conclude that these *numbers* are different..." instead of "... conclude that these *samples* are different." - I mention this because it suggests to me that such sloppiness is *not* pardonable; that the people who care most about the 'actual outcomes' are the ones who are least able to keep in mind the technical reservations. That's my advice, for what it is worth -- I have never taught any regular course. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at http://jse.stat.ncsu.edu/ =
Re: One-tailed, two-tailed
[ posted and e-mailed.] On Sat, 29 Dec 2001 16:46:10 -0500, [EMAIL PROTECTED] (Stan Brown) wrote: [... ] > > [The student is led to enter two sets of unpaired figures into > Excel. They represent miles per gallon with gasoline A and gasoline > B. I won't give the actual figures, but here's a summary: > > A: mean = 21.9727, variance = 0.4722, n = 11 > B: mean = 22.9571, variance = 0.2165, n = 14 > > The question is whether there is a difference in gasoline mileage. [snip, about variances.] > > "What's your conclusion about the difference in gas mileage?" > [Answer: At significance level 5%, previously selected, there is a > difference between them.] > > Now we come to the part I'm having conceptual trouble with: "Have > you proven that one gas gives better mileage than the other? If so, > which one is better?" > > Now obviously if the two are different then one is better, and if > one is better it's probably B since B had the higher sample mean. I want to raise an eyebrow at this earlier statement. We should not overlook the chance to teach our budding statisticians: *Always* pay attention to the distinction between random trials or careful controls, on the one hand; and grab-samples on the other. [Maybe your teacher asked the question that way, in order to lead up to that in class?] The numbers do not *prove* that one gas gives better mileage; the mileage was, indeed, better for one gas than another -- for reasons yet to be discussed. Different cars? drivers? routes? > But are we in fact justified in jumping from a two-tailed test (=/=) > to a one-tailed result (>)? > > Here we have a tiny p-value, and in fact a one-tailed test gives a > p-value of 0.0001443. But something seems a little smarmy about > first setting out to discover whether there is a difference -- just > a difference, unequal means -- then computing a two-tailed test and > deciding to announce a one-tailed result. Another small issue. Why did the .00014 appear? In clinical trials, we observe the difference and then we do attribute it to one end. But it is not the convention to report the one-tailed p-level, after the fact. I think there are editors who would object to that, but that is a guess. Also, for various reasons, our smallest p-level for reporting is usually 0.001. > > Am I being over-scrupulous here? Am I not even asking the right > question? Thanks for any enlightenment. > > (If you send me an e-mail copy of a public follow-up, please let me > know that it's a copy so I know to reply publicly.) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Is this how you would have done it?
On Sat, 22 Dec 2001 09:02:23 -0500, "Ralph Noble" <[EMAIL PROTECTED]> wrote: ... > A local newspaper asked its readers to rank the year's Top 10 news stories > by completing a ballot form. There were 10 choices on all but one ballot > (i.e. local news, sports news, business news, etc.), and you had to rank > those from 1 to 10 without duplicating any of your choices. One was their > top pick, 10 their lowest. Only one ballot had more than 10 choices, because > of the large number of local news stories you could choose from. > > > > I would have thought if you only had 10 choices and had to rank from 1 to > 10, then you'd count up all the stories that got the readers' Number One > vote and which ever story got the most Number One votes would have been > declared the winner. [ ... ] I have read three good responses. I want to mention that what you describe is just like the polls used in college sports, where the coaches or media each vote (coaches do one poll; media do another) for "Who is number 1."And they typically do report what you ask for, the number of #1 votes, in addition to the total (which does not have to agree). Thinking of other ratings with rankings: Is it "Places rated almanac"? - that annually lists numbers for 120 or 250 American cities. They do another simple averaging of ranks, across their 10 or so categories. I remember some public discussion of how arbitrary that was, and how it tends to reward 'consistent mediocrity.' (The first time they did this, Pittsburgh was near the top, so I have noticed later discussion.) That discussion also pointed out that the results were *usually* not *greatly* different if you chose another weighting system. And the better advice was that any interested person should look at the separate categories, and choose the few that matter to themselves. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: How ro perform "Runs Test"??
On Sun, 23 Dec 2001 23:48:58 GMT, "Jim Snow" <[EMAIL PROTECTED]> wrote: > > "Glen" <[EMAIL PROTECTED]> wrote in message > [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > > [EMAIL PROTECTED] (Chia C Chong) wrote in message > news:<[EMAIL PROTECTED]>... > > > I am using nonlinear regression method to find the best parameters for > > > my data. I came across a term called "runs test" from the Internet. It > > > mentioned that this is to determines whether my data is differ > > > significantly from the equation model I select for the nonlinear > > > regression. Can someone please let me know how should I perform the > > > run tests?? > > > > You need to use a runs test that's adjusted for the dependence in the > > residuals. The usual runs test in the texts won't apply. > > > > Glen > > I always understood that the runs test was designed to detect systematic > departures from the fitted line because some other curve fitted the data > better. In this context, it is a test for dependence of residuals. > > There is a discussion of this at > http://216.46.227.18/curvefit/systematic_deviation.htm > > Any elementary text in Non-parametric Methods in statistics will > give an example. Well, the residuals are always *dependent*, to the extent of p/n (# variables divided by N). That is the Expectation. So they are *not* i.i.d, which is an assumption. Thus: the runs test is an approximation which is inadequate for large ratios of p/n -- It is nice for the stat-pack to explain the runs-test, but not-so-nice that it fails to mention the other detail. Draper and Smith's book on regression mention that the runs test will be approximate, since the expectation is not independent. You can also google-search on <"Durbin-Watson" "runs test">, and click on the lectures ... or whatever appeals most to you. The D-W test is awkward enough to *test* that you don't wonder why people should look for an easier option. Several textbooks that I just looked at seem to be satisfied with recommending that you eye-ball your residuals in several plots - without doing tests. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What is the formulae for generalised Gaussian CDF???
On 15 Dec 2001 06:41:29 -0800, [EMAIL PROTECTED] (Chia C Chong) wrote: > Can somebody tell mw what is the formulae for the CDF of generalised > Gaussian distribution or where can I refer to??? > http://amp.ece.cmu.edu/Publication/Trista/ACMSecurity_Trista.pdf - has an equation. That equation cites [16] J. R. Hernández, M. Amado, and F. Pérez-González, "DCT- Domain Watermarking Techniques for Still Images: Detector Performance Analysis and a New Structure". IEEE Transactions on Image Processing, 9(1), 55-68. That was one of the first files I looked at after a www.google.com search on "generalized gaussian"; so, you can probably find others. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Standardizing evaluation scores
On Wed, 19 Dec 2001 02:11:01 GMT, Doug Federman <[EMAIL PROTECTED]> wrote: > I have a dilemma which I haven't found a good solution for. I work with > students who rotate with different preceptors on a monthly basis. A > student will have at least 12 evaluations over a year's time. A > preceptor usually will evaluate several students over the same year. > Unfortunately, the preceptors rarely agree on the grades. One preceptor > is biased towards the middle of the 1-9 likert scale and another may be > biased towards the upper end. Rarely, does a given preceptor use the 1-9 > range completely. I suspect that a 6 from an "easy" grader is equivalent > to a 3 from a "tough" grader. Huge rater-differences? that implies that you can't keep the original 1-9 and be fair. Doesn't it? And you can't trust the labels stuck with the original 1-9. > > I have considered using ranks to give a better evaluation for a given > student, but I have a serious constraint. At the end of each year, I > must submit to another body their evaluation on the original 1-9 scale, > which is lost when using ranks. > > Any suggestions? The original 1-9 seems to have no rational anchor, if the raters can vary between by 3 points on a standard. So, you average your ranks; sort the students that way; compute and list their raw averages along with rank averages. If extreme HIGH scores or LOW scores are important, you could avoid ranks. (a) Simply rescore each preceptor to have an average of 5, or (b) normalize each preceptor by mean and standard deviation. Assume ranking. I think, from that step, I might assign the mid-average to the mid-rank -- this would have the effect of anchoring my eventual, final report to the same range the raters used. (Assuming this matters.) Then I would work my way to the extremes, 'grading by the curve.' -- If you are trying to respect the labels on the 1-9 Likert, you have to consider what they actually *say*. You might justify giving the best student a '9' despite an average of 6.9 if the label says "best, consistent performance." -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: chi square validity?
On Tue, 18 Dec 2001 14:19:34 + (UTC), [EMAIL PROTECTED] (Benjamin Kenward) wrote: > > Let's say you have a repeatable experiment and each time the result can be > classed into a number of discrete categories (in this real case, seven). > If a treatment has no effect, it is known what the expected by chance > distribution of results between these categories would be. I know that a > good test to see if a distribution of results from a particular treatment > is different to the expected by chance distribution is to use a > chi-squared test. What I want to know is, is it valid to compare just one > category? In other words, for both the obtained and expected > distributions, summarise them to two categories, one of which is the > category you are interested in, and the other containing all the other > categories. If the chi-square result of the comparison of these categories > is significant, can you say that your treatment produces significantly > more results in particularly that category, or can you only think of the > whole distribution? Mathematically, the statistical test is okay: There is no problem if you decided at the outset that 7 categories should be scored as D-and-not-D, so you would do a 2x2 contingency table test. Other inferences, of course, are more problematic. "Multiple-tests." Deciding on a test based on the outcomes is a form a cheating in the hypothesis testing, if you don't take that into account in the reporting of it. If your overall test is significant -- with 6 d.f., I think -- then it is somewhat conventional to look at the separate contributions by cell, without being too shy. If the overall test is *not* that happy, then you ought to state that, and offer further guesses as purely exploratory or suggestive numbers. Then you can describe one cell's contribution "versus the others." -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Transaction Anomaly on 9/11????
On Mon, 17 Dec 2001 15:20:55 GMT, [EMAIL PROTECTED] (J. Williams) wrote: > I recently read about German-based Convar helping cos. in NYC uncover > the facts surrounding the unusual surge (both in volume and amounts) > in financial transactions during & immediately preceding the WTC > disaster. Convar is using a laser scanning technology to recover data [ snip ] Do you have a citation for your "unusual surge"? In my daily newspapers (NY Times, for one), I read the opposite -- there was no indication at all that the plans of the terrorists had leaked out to speculators, of any ilk. And investigators looked. I'm sorry, but this post reads to me like a pandering to hostile paranoia, and that annoys me; so I won't say anything more. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Basics
On Fri, 14 Dec 2001 09:22:26 +1100, "colsul" <[EMAIL PROTECTED]> wrote: > Read the request closely as answering newsgroup queries without > understanding what is said can make you look, no, confirm you are stupid. In Sensitivity-group country, we try to say (if confrontation is necessary) that an *action* is stupid or poorly considered, not that a person is stupid. (That is usually enough to make one's point... without getting embarrassed by over-ambitious flaming) In sci.stat.edu, we usually post the content that we are replying to, before we add comments; and we might criticize in some fashion, but we would never call a regular post-er 'stupid', who has offered dozens or hundreds of useful comments in the past. Before posting to a particular Internet group, it is well-advised to read a number of posts there, to pick up on the acceptable content and style. Besides all that, I have to say: from what I made of your question, I agree 100% with Glen's response. He gave a (relatively) gentle warning about a course of action. As a matter of fact, I find it hard to *imagine* cramming (which you ask about) for a job interview, unless it was following the advice, 'Learn about the job/ company.' But that's not something to look up in a textbook. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Evaluating students: A Statistical Perspective
On 7 Dec 2001 14:24:17 -0800, [EMAIL PROTECTED] (Dennis Roberts) wrote: > At 08:08 PM 12/7/01 +, J. Williams wrote: > >On 6 Dec 2001 11:34:20 -0800, [EMAIL PROTECTED] (Dennis Roberts) wrote: > > > > >if anything, selectivity has decreased at some of these top schools due to > > >the fact that given their extremely high tuition ... > > > i was just saying that IF anything had happened ... that it might have gone > down ... i was certainly not saying that it had ... > > but i do think that it could probably not get too much more selective ... > so it probably has sort of stayed where it has over the decades ... so if > grade inflation has occurred there it would not likely be due to an > increased smarter incoming class > (In the NY Times) At Harvard in particular, the interviewees claimed that the present freshmen had notably better SATs than those of a generation ago -- There are not nearly so many people with so-so scores (alumni offspring?), and a quarter of the class now has SATs that are perfect 1600, or nearly that (?no explanation of what 'nearly' means). I go along with the notion that, in the long run, if there is to be special meaning to being an "honors graduate" from Harvard, it can't mean "top 75% of the class". (I think that is what someone reported, somewhere.) I remember reading, years ago, that the Japanese school trajectory differed from ours -- they learnt a lot before college, and college was a long party before starting a career. (This was a few years ago.) Their life-long success was pre-determined largely by which-university accepted them; it sounded like the old-school-tie was a huge social asset. Reportedly, that was why their high school students worked so hard on cram courses and extra studying; college was 4 years of party. - Since they are sliding away from lifetime employment, etc., I wonder if the educational system is becoming more flexible and technocratic, too. Are our systems converging? -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: When to Use t and When to Use z Revisited
On Mon, 10 Dec 2001 12:57:29 -0400, Gus Gassmann <[EMAIL PROTECTED]> wrote: > Art Kendall wrote: > > (putting below the previous quotes for readability) > > > Gus Gassmann wrote: > > > > > Dennis Roberts wrote: > > > > > > > this is pure speculation ... i have yet to hear of any convincing case > > > > where the variance is known but, the mean is not > > > > > > What about that other application used so prominently in texts of > > > business statistics, testing for a proportion? > > > the sample mean of the dichotomous (one_zero, dummy) variable is known, It > > is the proportion. GG > > Sure. But when you test Ho: p = p0, you know (or pretend to know) the > population variance. So if the CLT applies, you should use a z-table, no? > That is the textbook justification for chi-squared and z tests in the sets of 'nonparametric tests' which are based on rank-order transformations and dichotomizing. The variance is known, so the test statistic has the shorter tails. (It works for ranks when you don't have ties.) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What usually should be done with missing values when I am conducting a t test or other tests?
On Fri, 07 Dec 2001 04:59:46 GMT, Richard J Burke <[EMAIL PROTECTED]> wrote: > jenny wrote: > > > What should I do with the missing values in my data. I ned to perform > > a t test of two samples to test the mean difference between them. > > > > How should I handle them in S-Plus or SAS? > > > > Thanks. > > JJ > > If you are doing paired tests, then the pairs with missing values will > have to be ignored; otherwise, you will simply have two samples of > different sizes. Going a step further -- The awkward dataset has *some* overlap from test-1 to test-2, and most Subjects exist just in one set or the other. But there are enough pairs to test. Then you have the paired test, and the unpaired test; and you can test assumptions about whether either set might me missing non-randomly. (And if everything looks kosher, you can sum the two tests.) But you should try to ask everyone involved, "Why and how do you happen to have data like these?" - If there is a Why, you need to find out Why; else, you want to educate them to do better next time. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Evaluating students: A Statistical Perspective
Just in case someone is interested in the Harvard instance that I mentioned -- while you might get the article from a newsstand or a friend -- On Sun, 02 Dec 2001 19:19:38 -0500, Rich Ulrich <[EMAIL PROTECTED]> wrote: [ ... ] > > Now, in the NY Times, just a week or two ago. The > dean of undergraduates at Harvard has a complaint > about grade inflation. More than 48% of all undergraduate > grades last year were A. (In 1986, it was only 34% or so.) > Only 6% or present grades were C or D or F. > > The dean has asked the faculty to discuss it, which is > as much as she can do. I don't know: Would the A's > emerge as scores on-a-curve, or are the lessons so > easy that all the answers are right? [ snip, rest] Section A of the NY Times on Wed., Dec 5, had another article (page 14) and a column (page 21). There were specific comments *contrary* to some obvious notions of grade "inflation" as an arbitrary and bad thing: some were presented as opinion, and other as apparent fact. Recent Harvard students have higher SATs than ever. Students at a particular level (of SAT, or otherwise) supposedly are performing better. The Dean of Harvard College (a subunit, I think) says that his students (in computer science) handle some previously-tough problems much more easily. [ And I wonder, Is that peculiar to cs.] Someone else was quoted, that the performance needed for an A had not changed. Amongst the commentary in the column - Comments on educational research: Good students (some research says) learn more if top grades are kept lower, but lower grading can discourage poorer students and increase dropout rates. - Both effects are easy to imagine, somewhere, sometime, -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Important notice about Final Exam on Thursday
I have a question concerning one detail - On 5 Dec 2001 10:04:38 -0800, [EMAIL PROTECTED] (Dennis Leitner) wrote: > [ ... ] To use multiple regression > you need to code the independent variables: Gender has 2 level; so > there needs to be 1 coded vector. SES has 3 levels so you need 2 coded > vectors. And then there will be 2 coded vectors for the interaction. A > total of 5 vectors. So far so good. Most of you did that. But then to > test for the Gender Effect you need to get the R-squared (or SS) from a > FULL MODEL with all 5 vectors and a RESTRICTED MODEL leaving out the > Gender Variable (so only 4 vectors remain). And THEN you use THE F > formula (R-squared full - R-sqaured restricted etc) to calculate the F. > You cannot read the F from the printout because it is NOT over and > above. [ ... ] I know, I can't be sure, since I don't have any version of the Q and A or printout in hand. But I can't imagine what else is being discussed The F-test on a 1-df hypothesis is given in ANOVA or Regression by the test on the difference in R-squared two models. And that is EXACTLY the same as the F-test on the beta in the fuller model; that was (after all) a partial-beta. What am I missing? -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: When does correlation imply causation?
On 5 Dec 2001 08:52:41 -0800, [EMAIL PROTECTED] (Dennis Roberts) wrote: > correlation NEVER implies causation ... That is true - in the strong sense, and - in formal logic, and - as a famous quotation among researchers. (And, reported as wrongly contrasted to 'ANOVA'.) Or, correlation always implies causation - intuitively, or - naively, or - in the weak sense. (I admit it -- I see a sexy correlation and 'implications' cross my mind.) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What usually should be done with missing values ...
On Tue, 04 Dec 2001 17:39:53 GMT, Jerry Dallal <[EMAIL PROTECTED]> wrote: > Rich Ulrich wrote: > > [ ... ] > > I don't see much difference. "Identifying predictors" by regression > > analyses -- what is that advice supposed to mean? The criticisms > > of stepwise selection say that it gives you the wrong variables, not > > 'merely' (as if that were trivial) the wrong weights. > > > > Am I missing something? (I am not totally against stepwise; > > just, mostly.) JD > > There are two issues: determining the right set of variables and > predicting the response. Stepwise can be deadly for the former > without, I believe, being too bad for the latter. I'm willing to > recant if someone with authority claims otherwise. I try to avoid the word "predict" when I am fitting parameters to describe a set of data or retrieve a formula - "post-dicting." That's okay, if that is what you mean. You are just talking about arriving at the minimum squared error in the fit. Oh, someone may ask, why not use them all? In my experience, I knew that there were supposed to be a certain number of variables in the equation; so I hoped to recover the actual formula, by restricting the variables. Starting out with pretty good hints as to variables and transformations, and some exact, tabled-up, results, I once used stepwise selection to recover a formula for the wind-chill index. Another time, I recovered the NFL formula for rating quarterbacks (and detected, with pretty high confidence, an error in the raw numbers). Right ballpark? -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: KS Statistic and adjustments
On Tue, 4 Dec 2001 15:57:35 + (UTC), "Steve Jones" <[EMAIL PROTECTED]> wrote: > Hi everyone, > I'd like your opinion on the following. > I am interested in computing the KS Statistic. > I have a binary choice variable --Buy or Not Buy > (coded 1 or 0). Using logistic regression I > have calculated the probability of buying. Kolmogorov-Smirnov? You won't do that on a binary variable. What are you planning to use it for? Having a scaling that is rational is usually more important than having one that is strictly normal; and where it matters, anyway, is for the tests. But you should be aware that you do *not* rely on the simple, overall ANOVA tests that you generate after "weighting" to match a population. -- If the weighting is not so slight as to make no practical difference, you have to use programs like SUDAAN for testing the contrasts. > > In preparing the data for regression, I did > a sort of stratified sampling, and generated > a weight variable (W) that will be applied to > the sample so that the sample will resemble > the true population distribution. > The question: After already included the > weight in computing the probability of buying, > is it necessary to use the weight again in > calculating the KS stat. > > Any arguments for or against? If you want the KS stat as a TEST about your sample, you don't want weighting. If you want an empirical, descriptive measure, maybe you would weight -- but WHO ever used the KS as a descriptive measure? I don't remember that. I think you are over-valuing normality and over-valuing the testing of normality. Also, KS is not particularly great for 'pre-testing for normality'. Assuming that is your purpose, I think Wilks-Shapiro is a better test. And a box-and-whisker plot shows more about bad outliers (ones that you need to make decisions about). -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What usually should be done with missing values ...
On Mon, 03 Dec 2001 20:57:33 GMT, Jerry Dallal <[EMAIL PROTECTED]> wrote: > Rich Ulrich wrote: > > > > On 2 Dec 2001 16:49:25 -0800, [EMAIL PROTECTED] (Kevin C. Heslin) > > wrote: > > > > > Jenny -- here's a way to impute continuous variables using SAS: > > > > > > Regression analysis is performed on a continuous variable until significant > > > predictors of the continuous variable are identified. The parameter estimates RU> > > > - "until significant predictors of the continuous variable are > > identified." > > > > That does limit the exercise to being strictly exploratory. > > > > You can see my stats-FAQ (or groups.google search) > > for criticisms of stepwise selection. JD> > This is a bit different, though. The criticisms of stepwise > selection are directed toward the attempt to assess the contribution > of individual predictors. The predictions themselves should be I don't see much difference. "Identifying predictors" by regression analyses -- what is that advice supposed to mean? The criticisms of stepwise selection say that it gives you the wrong variables, not 'merely' (as if that were trivial) the wrong weights. Am I missing something? (I am not totally against stepwise; just, mostly.) > okay, except in this instance the subset of cases on which the > predictions are based will vary according to the set of variables > that is being considered. Hurts my head to think about it, but I > wouldn't be surprised if someone else said 'not to worry' if > "missing at random" is appropriate. People most often ought to drop variables that are highly missing, for reasons that are both logical and statistical. Sometimes you can salvage a question by recoding, for instance, < Yes/ No/ Missing> to <(definite) Yes/ Other >. Also, you can salvage a question by using two variables for test purposes: one that dummy-codes Missing as Yes/No; plus the original variable, with an arbitrary score for the Missing. You then consider the two variables together. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: What usually should be done with missing values ...
On 2 Dec 2001 16:49:25 -0800, [EMAIL PROTECTED] (Kevin C. Heslin) wrote: > Jenny -- here's a way to impute continuous variables using SAS: > > Regression analysis is performed on a continuous variable until significant > predictors of the continuous variable are identified. The parameter estimates - "until significant predictors of the continuous variable are identified." That does limit the exercise to being strictly exploratory. You can see my stats-FAQ (or groups.google search) for criticisms of stepwise selection. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Evaluating students: A Statistical Perspective
- I guess I am commenting on the statistical perspective, at least, to start with. On Fri, 23 Nov 2001 16:22:46 GMT, "L.C." <[EMAIL PROTECTED]> wrote: > The question got me thinking about this problem as a > multiple comparison problem. Exam scores are typically > sums of problem scores. The problem scores may be > thought of as random variables. By the central limit theorem, > the distribution of a large number of test scores should look > like a Normal distribution, and it typically (though not always) > does. Hence the well known bell curve. (Assume, for the sake > of argument that it holds here.) > The tests that I took throughout elementary and high school (grades 1-12) were certainly not 'normally distributed' or my scores would have had variation. Tests or homework: everything was truncated at the top. Consistency seems to be demanded by the system of putting in a 90-point cutoff for A. If the average were 75 or 80, a top score of 100 has 1/3 or 1/4 the effect of tossing in a bottom score of 0. (This does seem more appropriate as a 'meta-lesson' at the elementary level, that it would be at the college level. That is, I can see where it can be more appropriate to judge college students by best-performance, or end-of-term performance, where, for el-hi students, doing nothing but preparatory work, the *consistency* is part of the lesson.) I suppose that elementary textbooks *might* be calibrated so that hitting 90% of the answers correct is what ought to justify an A. - In that case, my teachers *might* have been judged by whether they assigned the easier questions or the tougher ones. The teachers did use some discretion in what they assigned. I do believe that my teachers did not put many really tough questions on a test or in homework BECAUSE that would 'bring down the average' -- and they did not have any other accepted model to use except the 90-point-A model, and they wanted to give a 'proper' proportion of A's. As a consequence, I arrived at college without ever learning to study. No course had ever placed any demand on me. The usual notion of 'grading on the curve' before college was that a teacher would give more A's than the 90-point model would justify; so we students tended to want it. But I don't remember it happening much at all. I first matriculated at Rice where the freshman classes were huge, and were pre-announced as being 'on the curve' -- though the cutoffs were not absolutely fixed in advance. After making test scores like 35 and 4 (out of 100), I could *learn* from the scoring. I don't think I would have learned to 'show my work' -- at least, not so quickly and dramatically -- if I had not had that 4 on a physics test. With the curve, and low, low averages, you do notice that a single *good* performance can outweigh several poor ones. So that is good. Now, in the NY Times, just a week or two ago. The dean of undergraduates at Harvard has a complaint about grade inflation. More than 48% of all undergraduate grades last year were A. (In 1986, it was only 34% or so.) Only 6% or present grades were C or D or F. The dean has asked the faculty to discuss it, which is as much as she can do. I don't know: Would the A's emerge as scores on-a-curve, or are the lessons so easy that all the answers are right? What is the purpose of an A, anyway? There are relatively few courses where the grade is matched against an outside criterion. When I think about it, I don't think the existence of the outside criterion is apt to matter. Law school -- state exams are a breeze in a few states, or they can be almost impossible. Different schools in the state will have different standards, I am sure. Medical boards, on the other hand, are national, and I think they are usually passed by decent students. But I don't think of the MDs I have known as being obsessed about the college boards that they, personally, passed some years ago. I think the professors in either sort of school have standards from elsewhere for the grades they assign. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Interpreting p-value = .99
On Sat, 1 Dec 2001 08:20:45 -0500, [EMAIL PROTECTED] (Stan Brown) wrote: > [cc'd to previous poster] > > Rich Ulrich <[EMAIL PROTECTED]> wrote in sci.stat.edu: > >I think I could not blame students for floundering about on this one. > > > >On Thu, 29 Nov 2001 14:39:35 -0500, [EMAIL PROTECTED] (Stan Brown) > >wrote: > >> "The manufacturer of a patent medicine claims that it is 90% > >> effective(*) in relieving an allergy for a period of 8 hours. In a > >> sample of 200 people who had the allergy, the medicine provided > >> relief for 170 people. Determine whether the manufacturer's claim > >> was legitimate, to the 0.01 significance level." > > >I have never asked that as a question in statistics, and > >it does not have an automatic, idiomatic translation to what I ask. > > How would you have phrased the question, then? Though I took this > one from a book, I'm always looking to improve the phrasing of > questions I set in quizzes and exams. [ snip, rest] "In a LATER sample of 200 ... relief for ONLY 170 people." The Query you give after that should not pretend to be ultimate. Are you willing to ask the students to contemplate that the new experiment could differ drastically from the original sample and its conditions? "Is this result consistent with the manufacturer's claim?" - you might notice that this sounds 'weasel-worded.' Well, extremely-weasel-worded *ought* to be suiting, for *proper* statistical claims from non-randomized trials. For the example: I would expect 15% of a grab-sample being treated for 'allergy' would actually have flu or a cold. Maybe the actual experiment was more sophisticated? "What do you say about this result? (include a statistical test using a nominal alpha=.01)." Also, "Why do I include the word 'nominal' here?" Ans: It means 'tabled value' and it helps to emphasize that it is hard to frame a non-random trial as a test; the problem is not presented with any such framing. Hope this seems reasonable. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: (none)
On 30 Nov 2001 07:34:14 -0800, [EMAIL PROTECTED] (Carl Huberty) wrote: "Need help. Would someone please give me a reference that discusses a chance value of a canonical correlation coefficient (or the square thereof). That is my question is: What is the expected value of R when rho is zero?" Have you tried the 2nd Edition of Cohen's book (1989 or 90) on power analysis? (The 1st and Revised editions don't have the final chapter or two.) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Interpreting p-value = .99
I think I could not blame students for floundering about on this one. On Thu, 29 Nov 2001 14:39:35 -0500, [EMAIL PROTECTED] (Stan Brown) wrote: > On a quiz, I set the following problem to my statistics class: > > "The manufacturer of a patent medicine claims that it is 90% > effective(*) in relieving an allergy for a period of 8 hours. In a > sample of 200 people who had the allergy, the medicine provided > relief for 170 people. Determine whether the manufacturer's claim > was legitimate, to the 0.01 significance level." > > (The problem was adapted from Spiegel and Stevens, /Schaum's > Outline: Statistics/, problem 10.6.) [ snip, rest ] "Determine whether the manufacturer's claim was legitimate, to the 0.01 significance level." I have never asked that as a question in statistics, and it does not have an automatic, idiomatic translation to what I ask. I can expect that it means, "Use a 1% test." But, for what? After I notice that the outcome was poorer than the claim, then I wonder if the test is, "Are these data consistent with the Claim? or do they tend to disprove it?" That seems some distance from the tougher, philosophical question of whether, at the time it was made, the claim was legitimate. That claim could NEVER, legitimately, have been *based* on these data. That is an idea that tries to intrude itself, to me, and makes it difficult to address the intended question. - By the way, it also bothers me that "90% effective" is apparently translated as "effective for 90% of the people." I wondered if the asterisk was supposed to represent "[sic]". -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: N.Y. Times: Statistics, a Tool for Life, Is Getting Short Shrift
On Fri, 30 Nov 2001 14:38:33 GMT, mackeral@remove~this~first~yahoo.com (J. Williams) wrote: > On 29 Nov 2001 07:03:13 -0800, [EMAIL PROTECTED] (Robert J. > MacG. Dawson) wrote: > > > >There is probably a reverse trend in the extreme tail; people probably > >overestimate the probability of getting (say) red fifty times in a row > >at Roulette simply because we don't have a good feel for really large > >and small numbers. > > I think you are right in that assumption. When I taught probability, > I found students had difficulty sensing numerical enormity or its > opposite in scientific notation or lots of zeros. Dealing with 16 > zeros to the right of the decimal, for example, becomes a complete > abstraction. - whereas, by contrast, we scientists can right it out with "scientific notation" with its powers of ten, and have something concrete, not abstract, because it is additive in the exponents.... or am I just making another complete abstraction of it? -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: t vs. z - recapitulation
- I am just taking a couple of questions in this note - On Thu, 29 Nov 2001 13:16:24 +0100, "Gaj Vidmar" <[EMAIL PROTECTED]> wrote: [ ... ] I saw some decent comments about the table; the table was not very useful. z is used with large N as 'sufficiently good' approximation for t. z is used when "variance is known" -- which is, in particular, when statistics are computed that were based on dichotomies or on ranks (if there were no ties). That's basically when variances are known. With big samples and ties, you are probably better off doing your rank-transformation, then using the F-test or t-test. Depending on the outliers, ANOVA (t-tests) might be useless, regardless of how big the sample *might* get -- that happens more often than careless analysts expect, when they don't watch for outliers. -- If you can't opt for a parametric transform, you might need to test after rescoring as 'ranks' or into categories (two or more). > > Note 2: t-test is very robust (BTW, is Boneau, 1960, Psychological Bulletin - not in *my* opinion (see below) - > vol. 57, referenced and summarised in Quinn and McNemar, Psychological > Statistics, 4th ed. 1969, with the nice introduction "Boneau, with the > indispesable help of an electronic computer, ...", still an adequate > reference?), whereby: > - skewness, even extreme, is not a big problem > - two-tailed testing increases robusteness - I was annoyed when I learned that those old-line authors would decide that a test was 'robust with two-tails' when it rejected 9% in one tail in 1% in the other. It felt somewhat like having been lied-to. I still disagree with that opinion. Fortunately, the *problem* of really-bad-p-values (in both directions) does not exist for equal N. Unfortunately, even for equal N, there *can* be a great loss in statistical power. So, you should be unhappy to see great skewness. But for either equal or unequal N, I am much happier if I can trust that the variable has been transformed to its proper metric; and if that metric does not have skewness, or heterogeneity of variance. If a variable *needs* to be transformed, please transform it. (But the 'need' issue is worth its own discussion.) > - unequal variances are a serious problem with unequal N's with larger > variance of smaller sample Oh, a problem with larger variance in EITHER sample. A different problem, one way or the other. Outliers cause loss of power for ANOVA, just as much as outliers screw up a mean -- If you see outliers, ask, Are you sure ANOVA is the right tool? > > Now, what to do if t is inadequate? - This is a whole complex issue in > itself, so just a few thoughts: > - in case of extreme skewness, Mann-Whitney is not a good alternative > (assumes symmetric distrib.), right? [ ... ] It assumes *same* distributions in two samples, not necessarily symmetric. What is your hypothesis going to be? What can you fairly conclude, if one sample occupies both ends of the distribution? -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Normal distribution
On Thu, 29 Nov 2001 14:37:14 -0400, Gus Gassmann <[EMAIL PROTECTED]> wrote: > Rich Ulrich wrote: > > > On Thu, 29 Nov 2001 15:48:48 +0300, Ludovic Duponchel > > <[EMAIL PROTECTED]> wrote: > > > > > If x values have a normal distribution, is there a normal distribution > > > for x^2 ? > > > > If z is standard normal [ that is, mean 0, variance 1.0 ] > > then z^2 is chi squared with 1 degree of freedom. > > > > And the sum of S independent z variates > > is chi squared with S degrees of freedom. > > Hold it! The sum of S independent z variates is normal. > The sum of the _squares_ of S independent z variates is > chi squared with S degrees of freedom. > > (But I am sure you knew that.) oops- make that < z^2 > for < z > of course - -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Normal distribution
On Thu, 29 Nov 2001 15:48:48 +0300, Ludovic Duponchel <[EMAIL PROTECTED]> wrote: > If x values have a normal distribution, is there a normal distribution > for x^2 ? If z is standard normal [ that is, mean 0, variance 1.0 ] then z^2 is chi squared with 1 degree of freedom. And the sum of S independent z variates is chi squared with S degrees of freedom. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: biostatistics careers
On 19 Nov 2001 20:56:58 GMT, [EMAIL PROTECTED] (A.J. Rossini) wrote: > >>>>> "RS" == Richard Seymann <[EMAIL PROTECTED]> writes: > > RS> And if I may muddy the waters even more, what is the > RS> difference between biostatistics and biometry? Dick > > Depends on which definition of "biometry" you are using. One > definition used to be an older name for what is now biostatistics (and > what might again become biometry, according to how some in the field > want to rename it again to make it more relevant to measurement and > design). That's a nice murky response, which shows how our language > is ill-defined. > > There are other definitions, which have little to do with statistics. I was a little startled by my google search. "Definition of biometry" gets the Cornell Department of Biometrics, and a couple of other things, among only 6 hits. "Definition of biometrics" returns dozens of references to security systems, fingerprints and retinal scans, etc. That is a definition that seems to be winning out, owing to its new, computerized, commercial potential. There were no hits that would make me think of the journal named Biometrika, which I remember as being highly mathematical. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: best inference
On Wed, 21 Nov 2001 10:24:54 -0600, Bill Jefferys <[EMAIL PROTECTED]> wrote: > In article <[EMAIL PROTECTED]>, > [EMAIL PROTECTED] (Dennis Roberts) wrote: > > #on this near holiday ... at least in the usa ... i wonder if you might > #consider for a moment: > # > #what is the SINGLE most valuable concept/procedure/skill (just one!) ... > #that you would think is most important when it comes to passing along to > #students studying "inferential statistics" > # > #what i am mainly looking for would be answers like: > # > #the notion of > # > #being able to do __ > > I'd say "the notion of inverse probability", but of course that's > because of where I am coming from :-) > I'd say, being able to re-frame the researcher's fuzzy curiosity as a "testable hypothesis" -- I think *that* is where you get the probability that is to be inverted. Good hypothesizing would not be so valuable except that it is rare. And there are so many people who are close to it, and need it. So it ought to be valuable, as a commodity in demand. As a commodity, at least, "the notion of inverse probability" is less valuable because it is out of reach. It is of concern, I think, to the people who have finally achieved a p-value, and wonder what to do next. (Actually, I don't have a quick opinion, but I thought Promoting Tests was a good way to pull Dennis's leg.) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: When Can We Really Use CLT & Student t
On 21 Nov 2001 10:18:01 -0800, [EMAIL PROTECTED] (Ronny Richardson) wrote: > As I understand it, the Central Limit Theorem (CLT) guarantees that the > distribution of sample means is normally distributed regardless of the > distribution of the underlying data as long as the sample size is large > enough and the population standard deviation is known. > > It seems to me that most statistics books I see over optimistically invoke > the CLT not when n is over 30 and the population standard deviation is > known but anytime n is over 30. This seems inappropriate to me or am I > overlooking something? [ snip, rest ] It seems to me that you have doubts which *might* be justifiable. Do you have a professor who is prone to glib generalizations? Do you have a lousy text? I do wonder if your textbooks actually say what you accuse them of, or if you are guilty of hasty overgeneralization. I have scanned textbooks in search of errors like those, but I hardly ever find any. Gross mis-statements tend to be in "handbooks" and in (unfortunate) interpretative articles by non-statisticians. (Can you cite "chapter and verse"?) -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: biostatistics careers
On Mon, 19 Nov 2001 08:07:33 -0500, Bruce Weaver <[EMAIL PROTECTED]> wrote: > > On Sun, 18 Nov 2001, Stan Brown wrote: > > > What _is_ "biostatistics", anyway? A student asked me, and I > > realized I have only a vague idea. > > > There was a thread on "biostatistics versus statistics" a couple years > ago, I think, but I was unable to find it at google groups. Maybe someone > out there saved some of it. groups.google.com on < biostatistics statistics > - I found a couple of notes, within the top 100. There were several comments on 23 Feb 2000 with the subject line, "re:biostatistics". They mention that medical background is important. And vocabulary. Also, as I vaguely remembered, I personally had answered a similar question, on 18 Feb 1998: === from my 1998 comment: - There are a couple of dozen or so U.S. universities that include a "Graduate School of Public Health." Here at the University of Pittsburgh, it is the GSPH that awards a degree in biostatistics. The course work for the degree does include courses which would not be required for "statistics" as I imagine it - epidemiology (chronic vs acute diseases), vital statistics, health services administration. [ ... ] === end of 1998 citation -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: [HELP NEEDED] What is the best technique to analyze the following experiment?
On 16 Nov 2001 09:34:52 -0800, [EMAIL PROTECTED] (S.) wrote: > I performed the following experiment: > > Each user (U) used several interfaces (I). Both U and I are to be > treated as random factors. For each U and I combination, time (T), > errors (E) and satisfaction (S) were measured. The data looks > something like: > > U I T E S > --- --- --- --- --- > U1I1100 1090 > U1I2200 2080 > U1I3300 3070 > U1I4400 4060 > U2I1102 1191 > U2I2198 1881 > U2I5500 5050 > U2I6600 6040 > . > . > . > etc. > > Please note that NOT all the users used all the interfaces. > > The question is: I wish to find the correlations between T, E and S > (viz., nullify the effects of U and I). What is the best statistical > method of doing this? I think something along the lines of Anova or For the little bit of data shown, the variable *I* has a huge effect, with R-squared of maybe 0.99 with each of the three variables, T, E and S, and that happens while I call it a continuous variable. So it would be just as important, with a waste of degrees of freedom, if it is used as categories; the table shows it coded as categories, I-1 to I-6. High R-squared puts you into the situation where subtle choices of model can make a difference. Is it appropriate to remove the effect of *I* by subtraction, or by division? - by category, or by treating it as continuous? > Variance Components should do that trick... I have SPSS, so any advice > on how to interpret the output will be most appreciated (please bear > in mind that I do not have a degree in statistics). If it is strictly correlation that you want, you can ask for the intercorrelations, while partial ling out the U and I variables. If *I* and U are to be partialled-out as categories, you can create a set of dummy variables, and partial-out those. The result that you get will *not* be robust against scaling variations (linear versus multiplicative, for instance). That is a consequence of the high R-squared and the range of numbers that you have. I suspect that the observed R-squared values might vary in a major way if you just change the raw data by a few points, too -- Note that prediction with an R-squared of 0.99 has *twice* the error of an R-squared of 0.995, and so on; that is approximately the same as the difference between 0.1 and 0.2, in certain, practical consequences. If it will please you to reduce the eventual intercorrelations to zero, a proper strategy *might* be to try alternative models to see if you can produce that result. Of course, in practice, it should be a great deal of help to know what the variables actually, are, and how they are scored, etc., to know what transformations are logical and appropriate. I suspect that data, as stated, leave out some conventional standardization, and so the observed correlations are mainly artifacts. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Usability of skills and knowledge
On 16 Nov 2001 09:16:08 -0800, [EMAIL PROTECTED] (Rolf Dalin) wrote: > In a discussion about the desired direction of development of intro > level statistics courses, a group where I am a member came to a > preliminary agreement that It is important to develop applied > statistics. I started to think about that concept in terms of main goals > of a course. The two main goals I suggest are > > 1. abilities to use statistics in scientific work > 2. ability to study statistics further > > So nr. 1 is the goal that concerns the "applied" part. So I went one > step further to try to express aspects of this, and I call it the > USABILITY aspects of the introductory course. I know that this word Relating to the Subject line, usability of skills: - an important job-skill: to not-screw-things up while getting something done. - important job-skills: to do right. - important skill: self-assessment, to know if the task *can* be done, or *has* been done pretty-much correctly and reliably. When it comes to having "basic skills in statistics," knowing the basic vocabulary is vital. However, the idea keeps coming back to me that people misuse the basic statistics because they don't understand the basics of logical and scientific inference. So, they need to re-study plane geometry. And read up on philosophy of science, and experimental design. Or join a debating society? > is used in the context of evaluating web applications. Now to my > questions: Can anybody direct me to an article or book that > discusses usability oin the context of knowledge? Is there a better > direction to follow when discussing the usefulness/applicability of > statistics skills and knowledge? -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Fw: We need your help!!!
On 12 Nov 2001 11:41:45 -0800, [EMAIL PROTECTED] (Carl Huberty) wrote: > It would be greatly appreciated if I could get references for the six topics > mentioned in the message below. I assume that Conover (1999) discusses the > first topic. But beyond that I am at a loss. Thanks in advance. > > Carl Huberty > - Original Message - > From: <[EMAIL PROTECTED]> > To: <[EMAIL PROTECTED]> > Cc: <[EMAIL PROTECTED]> > Sent: Sunday, November 11, 2001 9:10 AM > Subject: We need your help!!! > > > > Dear Sincere Prof. Huberty > > I hope that you are spending a happy time . > > I have a sincere friend from Egypt. He is going to prepare for his > > proposal and he is going to make his defense in the next few days. He > > asked our help to provide him with some information about: > > > > Equations used to make Non-parametric factorial > > ANOVA :- > > Bradley's collapsed and reduce test. > > Harwell-Serlin's L test. > > Blair-Sawilowsky's adjusted rank transform test. > > Puri-Sen test. > > Fawcett and Salter's aligned rank test. > > A few minutes of searching with google shows that these names and tests are not widely known. Several are mentioned in the same articles. It looks like someone needs the Proceedings of the Joint Statistical Meetings in Toronto, August 15-18, 1994. Harwell-Serlin's L test. http://seamonkey.ed.asu.edu/~behrens/edstat/newsletter_95.html Blair-Sawilowsky's adjusted rank transform test. Also Harwell-Serlin: http://lib.stat.cmu.edu/joint94/section-a.html http://lib.stat.cmu.edu/joint94/Abstracts/0486 Puri-Sen test. http://www.iuinfo.indiana.edu/HomePages/021999/text/quantifying.htm Fawcett and Salter's aligned rank test. see 0486, above. Several tests are mentioned in a poster session with abstracts at http://www.stat.ufl.edu/symposium/1999/nonpar/poster-abstracts.html "The Best Test for Interaction in Factorial ANOVA and ANCOVA Todd C. Headrick and Shlomo S. Sawilowsky Wayne State University" I wonder... Isn't an education, these days, supposed to develop the ability to go online and search for topics like these? -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Standard Deviation!
On Sun, 11 Nov 2001 01:30:27 +1100, "David Muir" <[EMAIL PROTECTED]> wrote: > Presently the Gaming Industry of Australia is attempting to define various > new 'definitions of Standard Deviation'...in a concept to define infield > metrics for the analysis of machines in terms which imply whether a machine Google - does not find "infield metrics" or ""infield observable" (below) and I find the terms baffling. That is part of what makes the question baffling to me. The "standard deviation" is also a poser. I think someone will have to cite the proposed law. I do see where regulated gambling and "probability" raise some problems. If slot machines are supposed to return a certain percentage by law, is there a control on each machine to assure that each machine, separately performs within certain limits? Does this mean that the "winning" has to be monitored, so that it can periodically be adjusted? - that would make the old-style sucker potentially correct, the one who insisted on keeping with their machine because it was 'coming due.' But if the odds are more realistic and independent, then some machine, out of a million sequences for all machines, will have a long, losing sequence that is a-million-to-one. > is being operated with respect to its defined percentage or in fact outside > its defined region - i.e.. Illegally manipulated. > > My understanding of the Standard Deviation metric does not fit the mixed > (confused) proposals of the industry. Therefore I ask if a suitable > mathematician may be available to examine the problem. > > The gaming industry seems to want the metric used in terms of a periodic > infield observable, my feeling is that it is inappropriate and another > method must be provided if possible. The first part is to confer the > inappropriate aspect of STDDEV to periodic observations. After which > appropriate methods are required! > > Here is the problem: > > Imagine a game with two components, the base game and a feature. During the > base game prizes from say 20 units to 5000 are awarded (the standard > deviation being well defined). During the base games at > statistically 1 in say 120 occurrence, a feature game occurs which uses the > same base game prizes though all are multiplied by 3 i.e. 60 to 15000 unit > prizes. > > What method should be used to define the standard deviation? > > For anyone able to provide a provable solution a monetary prize of at least > $250US is available. 3 times the payoff == 9 times the sum of squares. So if you know what the "standard deviation" is for 119 out of 120 occurrences, and you call that *variance* 1.0; and you replace variance #120 with 9.0; then you can see how the standard deviation is increased. But this is a silly problem to me because I don't make any sense out of 'standard deviation.' -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Z Scores and stuff
On Thu, 8 Nov 2001 18:31:57 +, Mark T <[EMAIL PROTECTED]> wrote: > Hi, > > What are the formulae for calculating the mean to z, larger proportion and smaller >proportion of a z-score (standardised score) on a standard normal distribution? I >know about tables listing them all, but I want to know how to work it out for myself >:o) Do you want the calculus, or just a numerical approximation? For starters, in my stats-FAQ, see http://www.pitt.edu/~wpilib/statfaq/gaussfaq.html - contents contributed by John D'Errico. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Testing for joint probability between 2 variables
On Tue, 30 Oct 2001 21:10:02 -, "Chia C Chong" <[EMAIL PROTECTED]> wrote: [ ... ] > > The observations were numbers. To be specified, the 2 variables are DELAY > and ANGLE. So, basically I am looking into some raw measurement data > captured in the real environment and after post-proceesing these data, I > will have information in these two domains. > > I do not know whether there are linearly correlated or sth else but, by > physical mechanisms, there should be some kind of correlation between them. > They are observed over the TIME domain. I don't think it has been answered yet, whether they are correlated because they are autocorrelated in a trivial way. What does it mean here -- or does it happen to signify nothing -- that observation is "over the TIME domain". That is, you have a real problem yet to be faced, if these are measured as "cumulative delay" and "cumulative angle". -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: ranging opines about the range
[ I have rearranged Zar's note.] After this one,
> >>> Harold W Kerster <[EMAIL PROTECTED]> 10/29/01 04:31PM >>>
> If you define the range as max - min, you get zero, not one. What
> definition are you using.
On 29 Oct 2001 16:11:15 -0800, [EMAIL PROTECTED] (Jerrold Zar)
wrote:
> I was referring to the definition that others on the list had proposed:
> max - min +1. It is NOT a definition with which I agree.
If {max - min + K} had been a formal proposal, it should have
stated that K will be "1" when the numbers are reported to
the nearest integer, but -- generally -- K should sensibly
reflect the precision of measurement-and-reporting.
I don't know who needs it in the real world most of the time,
but using K gives a better -- usually safer -- estimate when
you are using the range to estimate the standard deviation.
But.
Whenever {max-min} is small enough that K is a sizable
correction, someone needs to speak carefully about the 'range.'
To put it another way: If all the cases all are observed at
the same value, and it matters, then the audience really
does deserve to hear the details.
--
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html
=
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=
Re: Graphics & CORRESPONDENCE ANALYSIS
On Mon, 22 Oct 2001 17:38:04 +0200, christian <[EMAIL PROTECTED]> wrote: > Hello, > > ...have anybody experience or a good idea > how i can display the result from the correspondence analysis > in a better way like spss (i.e. excel ), because the visual performance > in my humble opinion is not the best !? If you are looking for a model to program after -- I liked the display of the old BMDP procedure, CA, as shown on page 660 (and 664) of the 1990 manual (Vol 2). Here is one entry to a US web-site that has a lot on CA. http://www.okstate.edu/artsci/botany/ordinate/glossary.htm It has been my impression (from google) that CA is more popular in European journals than in the US, so there might be better sites out there in a language I don't read. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Standardized Confidence Intervals
On 15 Oct 2001 07:44:33 -0700, [EMAIL PROTECTED] (Warren) wrote: > Dear group, > It seems to me that the one issue here is that when we > measure something, then that measure should have some > meaning that is relevant to the study hypotheses. > And that meaning should be interpretable so that the width > of the CI does have meaning...why would you want to estimate > the mean if it is "meaningless"? This reminds me that data analysts sometimes can help to make outcomes more transparent. "Scaled scores" of Likert-like items are intelligible when presented as Item-averages, instead of being Scale-totals for varying numbers of items. You can describe the relevant anchors for 2.5 versus 3.0, for group contrasts, change scores, or contrasts between different scales -- and not get into confusion of how many items were added for each total. I do think that standardized outcomes are usually appropriate to communicate the size of the outcome effect -- even when the measure is fairly well known. This discussion leads me to conclude that you absolutely need to describe your sample *more* thoroughly when you are stuck with using standardized measures to describe all your outcomes. How variable is this group? How extreme is this group and sample (if it is a clinical trial)? A small N is especially problematic, since you do want to show how narrowly (or not) it was selected. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Transformation function for proportions
On Fri, 19 Oct 2001 11:58:18 +1000, "Glen Barnett" <[EMAIL PROTECTED]> wrote: > > Rich Strauss <[EMAIL PROTECTED]> wrote in message > news:[EMAIL PROTECTED]... > > However, the arcsin transformation is for proportions (with fixed > > It's also designed for stabilising variance rather than specifically inducing > symmetry. > Does it actually produce symmetry as well? > > > denominator), not for ratios (with variable denominator). The "proportion > > of sentences in a number of texts that belong to a certain category" sounds > > like a problem in ratios, since the total number of sentences undoubtedly > > vary among texts. Log transformations work well because they linearize > > such ratios. > > Additionally for small proportions logs are close to logits, so logs are > sometimes helpful even if the data really are proportions. Logs also go > some way to reducing the skewness and stabilising the variance, though > they don't stabilise it as well as the arcsin square root that's specifically > designed for it. The transformation is okay but not great for proportions less than (say) 5%. Jez Hill followed up on a reference that gave him this summer, and posted further detail -- ====== from June 27, 2001, Jez Hill. Subject: Re: [Q ] transforming binomial proportions Newsgroups: sci.stat.math Rich Ulrich wrote in article [EMAIL PROTECTED]: > The fixed variance was the main appeal of the approximation, > "arcsin(sqrt(p))". [snip] > "A more accurate transformation for small n has been tabulated by > Mosteller and Youtz."[ Biometrika 48 (1961):433.] Thanks very much for that - it looks pretty good to me at n=500, 6<=np<=494 FYI: Following up on your reference, Mosteller and Youtz give the following formula from Freeman and Tukey [Ann. Math. Statist. 21(1950):607]. arcsin(sqrt( np/(n+1) ))/2 + arcsin(sqrt( (np+1)/(n+1) ))/2 which gives asymptotic variance 821/(n+0.5) "for a substantial range of p if n is not too small". I find that the improvement is quite significant, to the point where I would be quite happy to use it even for np=1, 2 or 3 at n=500, minor glitches in that region notwithstanding. === end of post -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Transformation function for proportions
On Wed, 17 Oct 2001 15:50:35 +0200, Tobias Richter <[EMAIL PROTECTED]> wrote: > > > We have collected variables that represent proportions (i. e., the > proportion of sentences in a number of texts that belong to a certain > category). The distributions of these variables are highly skewed (the > proportions for most of the texts are zero or rather low). So my Low proportions, and a lot at zero? There is no way you can transform to "symmetry" when there is a clump at one end and a long tail at the other. First thought: the dichotomy of None/Some sometimes contains most of the information that is useful. Dummy Var1. Related thought: "none" is sometimes a separate dimension from what is implicitly measured by the continuous values above zero. If that dimension does seem useful: Dummy Var2. > question is: Is there a function that transforms the proportions into > symmetrically distributed variables? And is there a reliable statistics > text that discusses such transformations? "Symmetry" might happen, and it is good to have for the sake of testing. However, describing a scientific model with meaningful parameters is a better starting point, and you can devise tests from there. I mean: it is useful if you have a "Poisson model with a Poisson parameter", say, at the stage of setting up a model. You might want to take the square root before you do testing, and you know that is appropriate for the Poisson; but the raw Poisson parameter is a number that is ordinarily additive. I have not seen many texts that tackle transformations in the abstract. Finney's classic text on bioassay has a few pages. Or, I think, Mosteller and Tukey, "Data Analysis and Regression." -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Help for DL students in doing assignments
- Mathematical logic is a lot tougher when your translation fails to properly distinguish "any" and "every" and "some." On 15 Oct 2001 07:18:43 -0700, [EMAIL PROTECTED] (Dr. Fairman) wrote: [ ... ] > 4.If you agree with item #3 (if not - please argue - why), it means that > you are also agree with the statement: > "every even is (in particular) sum of any two primes". > That's what you needed me to prove. Needed: "Every even number can be written as the sum of two primes" or For [each] even number S, there always exists two prime numbers K and L such that S= K+L. Stated by Fairman, as read by me (a native speaker of English), "If K and L are *any* two primes, and S is any even number, it is true that S = K+L ." That is true for arithmetic-modulo-2. Otherwise, not. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Are parametric assumptions importat ?
On 12 Oct 2001 11:14:54 -0700, [EMAIL PROTECTED] (Lise DeShea) wrote: > Re robustness of the between-subjects ANOVA, I obtained permission from Dr. > Rand Wilcox to copy three pages from his book, "New Statistical Procedures > for the Social Sciences," and place them on a webpage for my students. He > cites research showing that with four groups of 50 observations each and > population standard deviations of 4, 1, 1, and 1, the empirical Type I > error rate was .088, which is beyond Bradley's liberal limits on sampling > variability [.025 to .075]. You can read this excerpt at Well, I suggest that a variance difference of 16 to 1 practically washes out the usual interest in the means. Isn't that beyond the pale of the usual illustrations of what is robust? I may remember wrong, but it seems to me that Tukey used Monte Carlo with "10% contamination" of a sample, where the contaminant had excessive variances: 10-fold for the variances? What can you say about an example like that? Will a Box-Cox transformation equalize the variances? (no?) Is there a huge outlier or two? If not -- if the scores are well scattered -- all the extreme scores in *both* directions will be in that one group. And a "mean difference" will implicitly be determined by the scaling: That is, if you spread out the low scores (say), then the group with big variance will have the lower mean. > www.uky.edu/~ldesh2/stats.htm -- look for the link to "Handout on ANOVA, > Sept. 19-20, 2001." Error rates are much worse when sample sizes are > unequal and the smaller groups are paired with the larger sigma -- up to an > empirical alpha of .309 when six groups, ranging in size from 6 to 25, have > sigmas of 4, 1, 1, 1, 1, 1. > > The independent-samples t-test has an inoculation against unequal variances > -- make sure you have equal n's of at least 15 per group, and it doesn't > matter much what your variances are (Ramsey, 1980, I think). But the ANOVA > doesn't have an inoculation. > > I tell my students that the ANOVA is not robust to violation of the equal > variances assumption, but that it's a stupid statistic anyway. All it can > say is either, "These means are equal," or "There's a difference somewhere > among these means, but I can't tell you where it is." I tell them to move > along to a good MCP and don't worry about the ANOVA. Most MCP's don't > require a significant F anyway. And if you have unequal n's, use > Games-Howell's MCP to find where the differences are. Some of us don't like MCPs. We think that the overall test is not (or at least, not always) a bad way to start, if a person *really* can't be more particular about what they want to test. And if you have unequal Ns, you are stuck with one approximation or another, which has to be ugly when the Ns are too unequal; or else you are stuck with inconsistent statements, where the smaller difference in means is 'significant' but the larger one is not. (I am unfamiliar with Games-Howell's MCP.) Just another opinion. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
