Re: effect size/significance
I think, some other folks are being sloppy about effect sizes. "Power analysis for the social sciences" is a book that defines small, medium and large effects in terms that are convenient and *usually* appropriate for the *social sciences* -- it makes no pretenses that these are universally applicable. Similarly -- On Thu, 13 Sep 2001 18:17:54 -0500, jim clark <[EMAIL PROTECTED]> wrote: > Hi > > I found the Rosenthal reference that addresses the following > point: > > On 13 Sep 2001, Herman Rubin wrote: > > The effect size is NOT small, or it would not save more > > than a very small number of lives. If it were small, > > considering the dangers of aspirin, it would not be used > > for this purpose. > > At http://davidmlane.com/hyperstat/B165665.html, one finds: > > "Rosenthal (1990) showed that although aspirin cut the risk of a > heart attack approximately in half, it explained only .0011 of > the variance (.11%). Similarly, Abelson (1985) found that batting > average accounted for a trivial proportion of the variance in > baseball game outcomes. Therefore, measures of proportion of > variance explained do not always communicate the importance of an > effect accurately." Stripping down the verbiage, that says: "[M]easures of ... variance ... do not ... communicate ... the effect." That is exactly what Herman asserted. A two-fold effect is "moderate" in epidemiology, whether you need 500 or 10,000 subjects-per-group to detect it. A five-fold effect is large. In epi, these are measured by the odds ratio (which is also the Relative risk when the occurrence rates are low). Epidemiologists do not use R-squared as an effect measure because it varies 20-fold for the same 500 versus 10,000. R^2 is robust as a *test* but it is not robust where the "odds ratio" is robust, as a measure of effect size. > > The reference for Rosenthal is: > > Rosenthal, R. (1990). How are we doing in soft psychology? > American Psychologist, 45, 775-777. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
I remember I read somewhere about different effect size measures and now I found the spot: A book by Michael Oakes, U. of Sussex, "Statistical Inference" 1990. The measures were (xbar-ybar)/s, Proportion misclassified, r squared (biserial corr) and w squared (which I think means the same as Rsq adj in ordinary linear regression). I would rather talk about these things as measures of different aspects of a relationship between two variables. (A quantitative and a qualitative with two categories in Oakes' example.): Statistical effect, explanatory power and strength of relationship. (... even if one could be derived from another ...) Other aspects which could be added as pieces of information would be p-value of test of no relationship, real world effects, causal mechanisms, consistency, responsiveness ... (these last from a Mosteller and Tukey reference). If we teach this, I think it would be more obvious that one single printout doesnt tell the whole story. And I think it would be a good thing to acheeve. Anyway I would be happy to read comments about aspects of relationships, since I have only just started to think about it in this way. /Rolf > Mike Granaas wrote: > > I think that we might agree: I would say that studies need a clear a > > priori rational (theoretical or empirical) prior to being conducted. It > > is only in that context that effect sizes can become meaningful. If a > > Even then standardized effect sizes may not be very helpful. We need > to know much more information about the effect, the sensitivity of > our measurements and so on. > > Thom > > > = > Instructions for joining and leaving this list and remarks about > the problem of INAPPROPRIATE MESSAGES are available at > http://jse.stat.ncsu.edu/ > = = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Mike Granaas wrote: > I think that we might agree: I would say that studies need a clear a > priori rational (theoretical or empirical) prior to being conducted. It > is only in that context that effect sizes can become meaningful. If a Even then standardized effect sizes may not be very helpful. We need to know much more information about the effect, the sensitivity of our measurements and so on. Thom = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Dennis Roberts <[EMAIL PROTECTED]> wrote: : given a simple effect size calculation ... some mean difference compared to : that is ... can we not get both NS or sig results ... when calculated : effect sizes are small, medium, or large? : if that is true ... then what benefit is there to look at significance AT ALL effect size does not depend on the standard error of the estimate so one can get a large nonsignificant effect. Obviously you want some reason to believe that the large estimated effect is real. = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Hi I found the Rosenthal reference that addresses the following point: On 13 Sep 2001, Herman Rubin wrote: > The effect size is NOT small, or it would not save more > than a very small number of lives. If it were small, > considering the dangers of aspirin, it would not be used > for this purpose. At http://davidmlane.com/hyperstat/B165665.html, one finds: "Rosenthal (1990) showed that although aspirin cut the risk of a heart attack approximately in half, it explained only .0011 of the variance (.11%). Similarly, Abelson (1985) found that batting average accounted for a trivial proportion of the variance in baseball game outcomes. Therefore, measures of proportion of variance explained do not always communicate the importance of an effect accurately." The reference for Rosenthal is: Rosenthal, R. (1990). How are we doing in soft psychology? American Psychologist, 45, 775-777. Best wishes Jim James M. Clark (204) 786-9757 Department of Psychology(204) 774-4134 Fax University of Winnipeg 4L05D Winnipeg, Manitoba R3B 2E9 [EMAIL PROTECTED] CANADA http://www.uwinnipeg.ca/~clark = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Hi On 13 Sep 2001, Herman Rubin wrote: > jim clark <[EMAIL PROTECTED]> wrote: > >Or consider a study with a small effect size that is significant. > >The fact that the effect is significant indicates that some > >non-chance effect is present and it might very well be important > >theoretically or even practically despite the small effect size. > >The classic example of practical significance is the use of > >aspirin prophylactically to prevent cardio-vascular problems. > >The effect size is quite small, but the effect was significant in > >clinical studies and, given the large numbers of people involved, > >was of great practical importance (i.e., many lives were > >extended). > > The effect size is NOT small, or it would not save more > than a very small number of lives. If it were small, > considering the dangers of aspirin, it would not be used > for this purpose. I was surprised by Herman's response here. I was going by memory, but I thought that someone (Rosenthal perhaps?) had demonstrated that conventional effect size statistics for the effect of aspirin were indeed small. The great benefit comes from a small effect that is applied to many, many people. An effect of .01 applied to 1,000,000 people represents 10,000 lives. I'll try to track it down. Best wishes Jim James M. Clark (204) 786-9757 Department of Psychology(204) 774-4134 Fax University of Winnipeg 4L05D Winnipeg, Manitoba R3B 2E9 [EMAIL PROTECTED] CANADA http://www.uwinnipeg.ca/~clark = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
jim clark wrote: > > > Sometimes I think that people are looking for some "magic > bullet" in statistics (i.e., significance, effect size, > whatever) that is going to avoid all of the problems and > misinterpretations that arise from existing practices. I think > that is a naive belief and that we need to teach how to use all > of the tools wisely because all of the tools are prone to abuse > and misinterpretation. > > Spot on! Alan -- Alan McLean ([EMAIL PROTECTED]) Department of Econometrics and Business Statistics Monash University, Caulfield Campus, Melbourne Tel: +61 03 9903 2102Fax: +61 03 9903 2007 = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
RE: effect size/significance
On Thu, 13 Sep 2001, Paul R. Swank wrote in part: > Dennis said > > other than being able to say that the experimental group ... ON AVERAGE ... > had a mean that was about 1.11 times (control group sd units) larger than > the control group mean, which is purely DESCRIPTIVE ... what can you say > that is important? > > However, can you say even that unless it is ratio scale? Yes, well, Dennis was referring to a _difference_. When the underlying scale is interval, differences ARE ratio scale: zero means zero. -- Don. Donald F. Burrill [EMAIL PROTECTED] 184 Nashua Road, Bedford, NH 03110 603-471-7128 = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
In article <[EMAIL PROTECTED]>, jim clark <[EMAIL PROTECTED]> wrote: >Hi >On 12 Sep 2001, Dennis Roberts wrote: >> that is ... can we not get both NS or sig results ... when calculated >> effect sizes are small, medium, or large? >> if that is true ... then what benefit is there to look at >> significance AT ALL ... >What your table shows is that _both_ dimensions are informative. >That is, you cannot derive effect size from significance, nor >significance from effect size. This has to be drummed into the users of statistics. To illustrate why you need both, >consider a study with small n that happened to get a large effect >that was not significant. The large effect should be "ignored" >as being due to chance. Only having the effect size would likely >lead to the error of treating it as real (i.e., non-chance. >Or consider a study with a small effect size that is significant. >The fact that the effect is significant indicates that some >non-chance effect is present and it might very well be important >theoretically or even practically despite the small effect size. >The classic example of practical significance is the use of >aspirin prophylactically to prevent cardio-vascular problems. >The effect size is quite small, but the effect was significant in >clinical studies and, given the large numbers of people involved, >was of great practical importance (i.e., many lives were >extended). The effect size is NOT small, or it would not save more than a very small number of lives. If it were small, considering the dangers of aspirin, it would not be used for this purpose. >So we need to be sensitive to both the magnitude and significance >of our effects, and we need to be duly cautious in the >interpretation of both. The way to take it all into account is to use the decision approach. This can be stated in one sentence: It is necessary to consider all consequences of the proposed action in all states of nature. Consistency applied to this states that one should use some weight measure on the utility evaluation of the consequences for the various states of nature, and sum or integrate. -- This address is for information only. I do not claim that these views are those of the Statistics Department or of Purdue University. Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399 [EMAIL PROTECTED] Phone: (765)494-6054 FAX: (765)494-0558 = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
In article <[EMAIL PROTECTED]>, jim clark <[EMAIL PROTECTED]> wrote: >Hi >On 13 Sep 2001, Rolf Dalin wrote: >> Hi, this is about Jim Clark's reply to dennis roberts. . >Sometimes I think that people are looking for some "magic >bullet" in statistics (i.e., significance, effect size, >whatever) that is going to avoid all of the problems and >misinterpretations that arise from existing practices. I think >that is a naive belief and that we need to teach how to use all >of the tools wisely because all of the tools are prone to abuse >and misinterpretation. This is what all those who have only taken methods courses, or in fact not studied it from a decision approach CAN do. The discussion of significance, confidence intervals, etc., ignores some aspects of the value of the decision, or else ignores some of the states of nature. The decision approach needs to be used with care, as the user's assumptions affect the decision taken, and this cannot be ignored; in some cases, the effects of some of the assumptions are small enough that they are not of much importance, but few users know which. -- This address is for information only. I do not claim that these views are those of the Statistics Department or of Purdue University. Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399 [EMAIL PROTECTED] Phone: (765)494-6054 FAX: (765)494-0558 = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
RE: effect size/significance
Dennis said other than being able to say that the experimental group ... ON AVERAGE ... had a mean that was about 1.11 times (control group sd units) larger than the control group mean, which is purely DESCRIPTIVE ... what can you say that is important? However, can you say even that unless it is ratio scale? OTOH, there is a two standard deviation difference, which is large enough to practically hit you over the head. The significance test is more important when the effect is smaller because small effects are much more likely to be chance events. Paul R. Swank, Ph.D. Professor Developmental Pediatrics UT Houston Health Science Center = = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
On Thu, 13 Sep 2001, Dennis Roberts wrote: > see the article that focuses on this even if they do report effect sizes ... ) > > what we need in all of this is REPLICATION ... and, the accumulation of > evidence about the impact of independent variables that we consider to have > important potential ... and not to waste our time and money on so many > piddly manipulations ... just for the sake of "getting stuff published" I was wondering when replication was going to get brought into this discussion. It's in all the textbooks, but we seem to forget it when it comes time to publish...can replications without mutation even be published? At the end of the day the individual study gives us data that is consistent, inconsistent, or ambigious about something. We add that bit of information to the balance scales until one of the arms goes down and stays down...then we truly know something. > > the recent push by apa journals and others to make submitters supply > information on effect sizes is, in my view, misplaced effort ... what > should be insisted upon in studies where the impacts of variables is being > investigated ... is a clear statement and rationale BY the researcher as to > WHAT size impact it would take to make a practical and important difference > ... I think that we might agree: I would say that studies need a clear a priori rational (theoretical or empirical) prior to being conducted. It is only in that context that effect sizes can become meaningful. If a study was done "just 'cause" then we will frequently not be able to make sense of the effect size measures. Michael > > if that ONE piece of information were insisted upon ... then all of us > would be in a much better position to evaluate results that are presented > > reporting effect sizes (greater than 0) does nothing to help readers > understand the functional benefit that might result from such treatments > > until we tackle that issue directly, we are more or less going around in > circles > > *** Michael M. Granaas Associate Professor[EMAIL PROTECTED] Department of Psychology University of South Dakota Phone: (605) 677-5295 Vermillion, SD 57069 FAX: (605) 677-6604 *** All views expressed are those of the author and do not necessarily reflect those of the University of South Dakota, or the South Dakota Board of Regents. = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
here are some data ... say we randomly assigned 30 Ss ... 15 to each condition and found the following: MTB > desc c1 c2 Descriptive Statistics: exp, cont Variable N Mean Median TrMean StDevSE Mean exp 15 26.13 27.00 26.00 4.96 1.28 cont15 21.73 22.00 21.85 3.95 1.02 MTB > twos c1 c2 Two-sample T for exp vs cont N Mean StDev SE Mean exp 15 26.13 4.96 1.3 cont 15 21.73 3.95 1.0 Difference = mu exp - mu cont Estimate for difference: 4.40 95% CI for difference: (1.04, 7.76) T-Test of difference = 0 (vs not =): T-Value = 2.69 P-Value = 0.012 < p value MTB > let k1=(26.13-21.73)/3.95 MTB > prin k1 Data Display K11.11392 simple effect size calculation other than being able to say that the experimental group ... ON AVERAGE ... had a mean that was about 1.11 times (control group sd units) larger than the control group mean, which is purely DESCRIPTIVE ... what can you say that is important? _ dennis roberts, educational psychology, penn state university 208 cedar, AC 8148632401, mailto:[EMAIL PROTECTED] http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
At 02:33 PM 9/13/01 +0100, Thom Baguley wrote: >Rolf Dalin wrote: > > Yes it would be the same debate. No matter how small the p-value it > > gives very little information about the effect size or its practical > > importance. > >Neither do standardized effect sizes. agreed ... of course, we would all be a whole lot better off if a researcher DEFINED for us ... a priori ... what size of effect he/she considered to be important ... and why that "amount" has practical benefit then we could evaluate (approximately) if he/she found something of practical importance (at least according to the researcher) but, what we get is an after the fact description of this which by the very nature of its post hocness ... is not really that helpful bringing into this discussion statistical significance only has relevance IF you believe that null hypothesis testing is of real value bringing effect size into the discussion only has relevance IF you know what effect it takes to have practical importance ... (and i have yet to see the article that focuses on this even if they do report effect sizes ... ) what we need in all of this is REPLICATION ... and, the accumulation of evidence about the impact of independent variables that we consider to have important potential ... and not to waste our time and money on so many piddly manipulations ... just for the sake of "getting stuff published" the recent push by apa journals and others to make submitters supply information on effect sizes is, in my view, misplaced effort ... what should be insisted upon in studies where the impacts of variables is being investigated ... is a clear statement and rationale BY the researcher as to WHAT size impact it would take to make a practical and important difference ... if that ONE piece of information were insisted upon ... then all of us would be in a much better position to evaluate results that are presented reporting effect sizes (greater than 0) does nothing to help readers understand the functional benefit that might result from such treatments until we tackle that issue directly, we are more or less going around in circles >Thom > > >= >Instructions for joining and leaving this list and remarks about >the problem of INAPPROPRIATE MESSAGES are available at > http://jse.stat.ncsu.edu/ >= _ dennis roberts, educational psychology, penn state university 208 cedar, AC 8148632401, mailto:[EMAIL PROTECTED] http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Hi On 13 Sep 2001, Rolf Dalin wrote: > Hi, this is about Jim Clark's reply to dennis roberts. > > I'm not sure how "both informative" gets translated into "neither > > very informative." Seems like a perverse way of thinking to me. > > Moreover, your original question was "then what benefit is there > > to look at significance AT ALL?" which implied to me that your > > view was that significance was not important and that effect > > size conveyed all that was needed. > > When using the information conveyed in the p-values and/or effect > size measures and/or decisions about some null hypothesis, in my > opinion there's only one place to look: effect size measures given > with CIs are informative. Significance alone gives you no clue to > whether an effect is of any practical importance in the real world > situation. I did not say anything about significance alone and (I thought) clearly indicated that it was important to consider such things as effect size, although effect sizes are no less difficult to use intelligently than significance tests (or confidence intervals). > > > the distinction between significant or not ... is based on an arbitrary > > > cutoff point ... which has on one side ... the notion that the null > > > seems as though it might be tenable ... and the other side ... the > > > notion that the null does not seem to tenable ... but this is not an > > > either/or deal ... it is only a matter of degree > > > > It was your table, but the debate would be the same if you put > > multiple rows with p values along the dimension. That is, what > > is the relative importance of significance (or p value) and > > effect size. > > Yes it would be the same debate. No matter how small the p-value it > gives very little information about the effect size or its practical > importance. Sorry I have to disagree. If you do not have a reasonably small significance level (or equivalently, a CI that does not show some evidence of non-overlap with a null effect), then any effort to study effect size or consider practical significance is wishful thinking. > When your data are on a scale which is arbitrary, not meters or > USDs, let's say you have constructed a scale from multiple items. > How do you define effect size? How can differences between means > be interpreted to be informative? To my knowledge effect size does not depend on the kind of measure. Many clinical scales in psychology, for example, are composite measures and effect sizes and practical significance can be computed and readily interpreted, albeit with some caution (not due to the nature of the measure but to the subtleties of all statistal methods). A clinician, for example, can indicate how much of a reduction on a measure of clinical depression would be clinically important. Sometimes I think that people are looking for some "magic bullet" in statistics (i.e., significance, effect size, whatever) that is going to avoid all of the problems and misinterpretations that arise from existing practices. I think that is a naive belief and that we need to teach how to use all of the tools wisely because all of the tools are prone to abuse and misinterpretation. Best wishes Jim James M. Clark (204) 786-9757 Department of Psychology(204) 774-4134 Fax University of Winnipeg 4L05D Winnipeg, Manitoba R3B 2E9 [EMAIL PROTECTED] CANADA http://www.uwinnipeg.ca/~clark = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Rolf Dalin wrote: > Yes it would be the same debate. No matter how small the p-value it > gives very little information about the effect size or its practical > importance. Neither do standardized effect sizes. Thom = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Hi, this is about Jim Clark's reply to dennis roberts. > On 12 Sep 2001, dennis roberts wrote: > > At 07:23 PM 9/12/01 -0500, jim clark wrote: > > >What your table shows is that _both_ dimensions are informative. > > >That is, you cannot derive effect size from significance, nor > > >significance from effect size. To illustrate why you need both, > > >consider a study with small n that happened to get a large effect > > >that was not significant. The large effect should be "ignored" > > >as being due to chance. Only having the effect size would likely > > >lead to the error of treating it as real (i.e., non-chance. > > > > or, another way to view it is that neither of the dimensions is very > > informative > > I'm not sure how "both informative" gets translated into "neither > very informative." Seems like a perverse way of thinking to me. > Moreover, your original question was "then what benefit is there > to look at significance AT ALL?" which implied to me that your > view was that significance was not important and that effect > size conveyed all that was needed. When using the information conveyed in the p-values and/or effect size measures and/or decisions about some null hypothesis, in my opinion there's only one place to look: effect size measures given with CIs are informative. Significance alone gives you no clue to whether an effect is of any practical importance in the real world situation. >... > > the distinction between significant or not ... is based on an arbitrary > > cutoff point ... which has on one side ... the notion that the null > > seems as though it might be tenable ... and the other side ... the > > notion that the null does not seem to tenable ... but this is not an > > either/or deal ... it is only a matter of degree > > It was your table, but the debate would be the same if you put > multiple rows with p values along the dimension. That is, what > is the relative importance of significance (or p value) and > effect size. Yes it would be the same debate. No matter how small the p-value it gives very little information about the effect size or its practical importance. When your data are on a scale which is arbitrary, not meters or USDs, let's say you have constructed a scale from multiple items. How do you define effect size? How can differences between means be interpreted to be informative? Cheers! /Rolf Dalin ** Rolf Dalin Department of Information Tchnology and Media Mid Sweden University S-870 51 SUNDSVALL Sweden Phone: 060 148690, international: +46 60 148690 Fax: 060 148970, international: +46 60 148970 Mobile: 0705 947896, intnational: +46 70 5947896 Home: 060 21933, intnational: +46 60 21933 mailto:[EMAIL PROTECTED] http://www.itk.mh.se/~roldal/ ** = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Hi On 12 Sep 2001, dennis roberts wrote: > At 07:23 PM 9/12/01 -0500, jim clark wrote: > >What your table shows is that _both_ dimensions are informative. > >That is, you cannot derive effect size from significance, nor > >significance from effect size. To illustrate why you need both, > >consider a study with small n that happened to get a large effect > >that was not significant. The large effect should be "ignored" > >as being due to chance. Only having the effect size would likely > >lead to the error of treating it as real (i.e., non-chance. > > or, another way to view it is that neither of the dimensions is very > informative I'm not sure how "both informative" gets translated into "neither very informative." Seems like a perverse way of thinking to me. Moreover, your original question was "then what benefit is there to look at significance AT ALL?" which implied to me that your view was that significance was not important and that effect size conveyed all that was needed. > of course we know that significance does not mean "real" and non > significance does not mean "chance alone" ... there is no way to decipher > one from the other based on our significance tests I included in parentheses after the "real" (i.e., non-chance). That is exactly what significance means ... the probability of this large a difference arising by chance (i.e., when Ho is true) is small. And non-significance does mean that the outcome had a reasonably high probability (i.e., chance) of occurring if the Ho were true. > the distinction between significant or not ... is based on an arbitrary > cutoff point ... which has on one side ... the notion that the null seems > as though it might be tenable ... and the other side ... the notion that > the null does not seem to tenable ... but this is not an either/or deal ... > it is only a matter of degree It was your table, but the debate would be the same if you put multiple rows with p values along the dimension. That is, what is the relative importance of significance (or p value) and effect size. > what if we juxtapose ... non significant findings with a large effect size > ... with significant results with a small effect size ... which of these > two would most feel "had most import"? Nonsignificant findings with a large effect size should be ignored (except perhaps as some indication that it might be worth replicating the study or gathering more observations to increase power). Significant results with a small effect size may or may not be important (e.g., the aspirin example), but at least there is little chance that they just occurred by chance. > surely, if both dimensions are seen as been equal parts of the overall > puzzle, then, both would seem to be more or less equal This is an over-simplification. I would say that significance has priority. Effect sizes for non-significant effects are meaningless except as described above. Given significance, one can turn attention to effect size, which itself is no simple matter. > but, if one opts for large effect size when results are not significant ... > then, this suggests that significance adds little if anything to the mix ... One should never trust effect sizes that are not significant, no matter how large. I would be very interested in seeing a case where this principle was violated to good effect. > however, if we opt for significance along with a small effect > size, then this suggests that significance is playing a more > important role in one's eyes Correct. The issue of whether the small effect size is an issue depends on numerous other factors. > the reality is too that effect sizes are, when categorized as small, > medium, and large ... again ... totally arbitrary ... which makes it even > harder still to make much sense of these ... just like it is difficult to > make much sense out of significance Are you thinking of retreating to some mountaintop and giving up on the world of statistics? > and finally, effect sizes do NOT say anything about importance OF the > effect ... merely some statement about the SIZE of the effect ... so, it > could very well be that for many independent variables ... a small effect > size has a much greater impact consequence than a large effect size ... > even when both are found to have significance on their side That is correct, as in the example I gave. > just to muddy the waters more And here I thought the waters were muddy all ready! Best wishes Jim James M. Clark (204) 786-9757 Department of Psychology(204) 774-4134 Fax University of Winnipeg 4L05D Winnipeg, Manitoba R3B 2E9 [EMAIL PROTECTED] CANADA http://www.uwinnipeg.ca/~clark
Re: effect size/significance
At 07:23 PM 9/12/01 -0500, jim clark wrote: >Hi > > >What your table shows is that _both_ dimensions are informative. >That is, you cannot derive effect size from significance, nor >significance from effect size. To illustrate why you need both, >consider a study with small n that happened to get a large effect >that was not significant. The large effect should be "ignored" >as being due to chance. Only having the effect size would likely >lead to the error of treating it as real (i.e., non-chance. or, another way to view it is that neither of the dimensions is very informative of course we know that significance does not mean "real" and non significance does not mean "chance alone" ... there is no way to decipher one from the other based on our significance tests the distinction between significant or not ... is based on an arbitrary cutoff point ... which has on one side ... the notion that the null seems as though it might be tenable ... and the other side ... the notion that the null does not seem to tenable ... but this is not an either/or deal ... it is only a matter of degree what if we juxtapose ... non significant findings with a large effect size ... with significant results with a small effect size ... which of these two would most feel "had most import"? surely, if both dimensions are seen as been equal parts of the overall puzzle, then, both would seem to be more or less equal but, if one opts for large effect size when results are not significant ... then, this suggests that significance adds little if anything to the mix ... however, if we opt for significance along with a small effect size, then this suggests that significance is playing a more important role in one's eyes the reality is too that effect sizes are, when categorized as small, medium, and large ... again ... totally arbitrary ... which makes it even harder still to make much sense of these ... just like it is difficult to make much sense out of significance and finally, effect sizes do NOT say anything about importance OF the effect ... merely some statement about the SIZE of the effect ... so, it could very well be that for many independent variables ... a small effect size has a much greater impact consequence than a large effect size ... even when both are found to have significance on their side just to muddy the waters more == dennis roberts, penn state university educational psychology, 8148632401 http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
Hi On 12 Sep 2001, Dennis Roberts wrote: > given a simple effect size calculation ... some mean difference compared to > some pooled group or group standard deviation ... is it not possible to > obtain the following combinations (assuming some significance test is done) > >effect size > small mediumlarge > > res NS > > res sig > > that is ... can we not get both NS or sig results ... when calculated > effect sizes are small, medium, or large? > > if that is true ... then what benefit is there to look at > significance AT ALL What your table shows is that _both_ dimensions are informative. That is, you cannot derive effect size from significance, nor significance from effect size. To illustrate why you need both, consider a study with small n that happened to get a large effect that was not significant. The large effect should be "ignored" as being due to chance. Only having the effect size would likely lead to the error of treating it as real (i.e., non-chance. Or consider a study with a small effect size that is significant. The fact that the effect is significant indicates that some non-chance effect is present and it might very well be important theoretically or even practically despite the small effect size. The classic example of practical significance is the use of aspirin prophylactically to prevent cardio-vascular problems. The effect size is quite small, but the effect was significant in clinical studies and, given the large numbers of people involved, was of great practical importance (i.e., many lives were extended). So we need to be sensitive to both the magnitude and significance of our effects, and we need to be duly cautious in the interpretation of both. Best wishes Jim James M. Clark (204) 786-9757 Department of Psychology(204) 774-4134 Fax University of Winnipeg 4L05D Winnipeg, Manitoba R3B 2E9 [EMAIL PROTECTED] CANADA http://www.uwinnipeg.ca/~clark = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: effect size/significance
At 04:04 PM 9/12/01 -0400, you wrote: if that is true ... then what benefit is there to look at significance AT ALL To get published, get tenure, and avoid having to live in a cardboard box in the park. Ha ha! Lise
effect size/significance
given a simple effect size calculation ... some mean difference compared to some pooled group or group standard deviation ... is it not possible to obtain the following combinations (assuming some significance test is done) effect size small mediumlarge res NS res sig that is ... can we not get both NS or sig results ... when calculated effect sizes are small, medium, or large? if that is true ... then what benefit is there to look at significance AT ALL _ dennis roberts, educational psychology, penn state university 208 cedar, AC 8148632401, mailto:[EMAIL PROTECTED] http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
