Re: SRSes
my previous remarks were about other sampling designs. I was comaring valid complex designs to SRS design and not non-sampling case selection. dennis roberts wrote: > my hypothesis of course is that more often than not ... in data collection > problems where sampling is involved AND inferences are desired ... we goof > far more often ... than do a better than SRS job of sampling > > 1. i wonder if anyone has really taken a SRS of the literature ... maybe > stratified by journals or disciplines ... and tried to see to what extent > sampling in the investigations was done via SRS ... better than that ... or > worse than that??? of course, i would expect even if this is done ... we > would have a + biased figure ... since, the notion is that only the > better/best of the submitted stuff gets published so, the figures for all > stuff that is done (ie, the day in day out batch), published or not ... > would have to look worse off ... > > 2. can worse than SRS ... be as MUCH worse ... as complex sampling plans > can be better than SRS??? that is ... could a standard error for a bad > sampling plan (if we could even estimate it) ... be proportionately as much > LARGER than the standard error for SRS samples ... as complex sampling > plans can produce standard errors that are as proportionately SMALLER than > SRS samples? are there ANY data that exist on this matter? > > == > dennis roberts, penn state university > educational psychology, 8148632401 > http://roberts.ed.psu.edu/users/droberts/drober~1.htm > > = > Instructions for joining and leaving this list and remarks about > the problem of INAPPROPRIATE MESSAGES are available at > http://jse.stat.ncsu.edu/ > = = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: SRSes
Dennis Roberts wrote: > > > but, we KNOW that most samples are drawn in a way that is WORSE than SRS ... > > thus, essentially every CI ... is too narrow ... or, every test statistic > ... t or F or whatever ... has a p value that is too LOW ... > > what adjustment do we make for this basic problem? The adjustment for design is done with weights to get the point estimates using regular software such as SPSS etc. To get the confidence estimates special software such as WESVAR, SUDAAN, or CPLEX is commonly used. Because the latter packages are not as user friendly in their presentation of results, I usually get the point estimates in SPSS, then I use WESVAR or SUDAAN and get both point and interval estimates. I use the point estimates from the latter packages as "navigational aids" to find the interval estimates in the output and to assure that I am getting the right computations. Some sampling designs include cluster sampling (random effects), some stratification (fixed effects), and some both. For those with stratification only, if there is any difference in the means (proportions) among the strata, usually the CIs will be too wide. For those with cluster sampling, usually the CIs will be too narrow. For those designs with both stratification and clustering, the CIs will be subject to both narrowing and widening, and only specilized software will tell the net effect. In addition, ratio, regression, or difference estimates may have narrower "true" CIs. = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
RE: SRSes
my hypothesis of course is that more often than not ... in data collection problems where sampling is involved AND inferences are desired ... we goof far more often ... than do a better than SRS job of sampling 1. i wonder if anyone has really taken a SRS of the literature ... maybe stratified by journals or disciplines ... and tried to see to what extent sampling in the investigations was done via SRS ... better than that ... or worse than that??? of course, i would expect even if this is done ... we would have a + biased figure ... since, the notion is that only the better/best of the submitted stuff gets published so, the figures for all stuff that is done (ie, the day in day out batch), published or not ... would have to look worse off ... 2. can worse than SRS ... be as MUCH worse ... as complex sampling plans can be better than SRS??? that is ... could a standard error for a bad sampling plan (if we could even estimate it) ... be proportionately as much LARGER than the standard error for SRS samples ... as complex sampling plans can produce standard errors that are as proportionately SMALLER than SRS samples? are there ANY data that exist on this matter? == dennis roberts, penn state university educational psychology, 8148632401 http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
RE: SRSes
Dennis Roberts writes: > most books talk about inferential statistics ... particularly those > where you take a sample ... find some statistic ... estimate some error > term ... then build a CI or test some null hypothesis ... > > error in these cases is always assumed to be based on taking AT LEAST a > simple random sample ... or SRS as some books like to say ... > > but, we KNOW that most samples are drawn in a way that is WORSE than SRS > thus, essentially every CI ... is too narrow ... or, every test > statistic ... t or F or whatever ... has a p value that is too LOW > > what adjustment do we make for this basic problem? Another thought provoking question from Penn State. In the real world, most people assess the deviation from SRS in a qualitative (non-quantitative) fashion. If the deviation is serious, then you consider it as a more preliminary finding or one that is in greater need of replication. If it is very serious, you totally disregard the findings from the study. The folks in Evidence Based Medicine talk about levels of evidence, and this is one of the things that they would use to select whether a study represents a higher or lower level of evidence. You probably do the same thing when you assess problems with non-response bias, recall bias, and subjects who drop out in the middle of the study. Typically you assess these in a qualitative fashion because it is so difficult to quantify how much these will bias your findings. You could argue that this represents the classic distinction between sampling error and non-sampling error. The classic CI is almost always too narrow, because it only accounts for some of the uncertainty in the model. We are getting more sophisticated, but we still can't quantify many of the additional sources of uncertainty. By the way, if you take non-SRS sample and then randomly allocate these patients to a treatment and control group, the CI appropriately accounts for uncertainty within this population, but you have trouble extrapolating to the population that you are more interested in. It's the classic internal versus external validity argument. I hope this makes sense and is helpful. Steve Simon, [EMAIL PROTECTED], Standard Disclaimer. STATS: STeve's Attempt to Teach Statistics. http://www.cmh.edu/stats Watch for a change in servers. On or around June 2001, this page will move to http://www.childrens-mercy.org/stats = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: SRSes
Dennis Roberts wrote: > but, we KNOW that most samples are drawn in a way that is WORSE than SRS ... > > thus, essentially every CI ... is too narrow ... or, every test statistic > ... t or F or whatever ... has a p value that is too LOW ... > > what adjustment do we make for this basic problem? We do it anyway! The real concern isn't that CIs are to narrow or that Ps are too liberal, but that they are completely irrelevant. If it's impossible to specify a sampling model, there's no formal basis for inference. (I'm ignoring randomized trials, which can be valid without being generalizable.) For better or worse, here's what I tell my students in an attempt at honesty... Sometimes the pedigree of a sample is uncertain, yet standard statistical techniques for simple random samples are used. The rationale behind such analyses is best expressed in a reworking of a quotation from Stephen Fienberg, in which the phrases (contingency table and multinomial have been replaced by survey and simple random): "It is often true that data in a [survey] have not been produced by a [simple random] sampling procedure, and that the statistician is unable to determine the exact sampling scheme which was used. In such situations the best we can do, usually, is to assume a [simple random] situation and hope that it is not very unreasonable." This does not mean that sampling issues can be disregarded. Rather, it says that in some instances we may treat data as though they arose from a simple random sample, barring evidence that such an approach is inappropriate. = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: SRSes
At 03:55 PM 7/24/01 -0400, Donald Burrill wrote: >Hi, Dennis! > Yes, as you point out, most elementary textbooks treat only SRS >types of samples. But while (as you also point out) some more realistic >sampling methods entail larger sampling variance than SRS, some of them >have _smaller_ variance -- notably, stratified designs when the strata >differ between themselves on the quantity being measured. sure ... i know that (then i said) ... but, we KNOW that most samples are drawn in a way that is WORSE than SRS and you responded >I don't think _I_ know this. I know that SOME samples are so drawn; >but (see above) I also know that SOME samples are drawn in a way that >is BETTER than SRS (where I assume by "worse" you meant "with larger >sampling variance", so by "better" I mean "with smaller sampling >variance"). i think we do know this ... if you enumerate all the situations you know of where sampling from some larger population has been done ... i would bet a dollar to a penny that ... the sampling plan is WORSE than SRS ... so, i would suggest that the NORM is worse ... the exception is SRS or better i don't think books spend nearly enough time ... on the fact that most day in day out samples are taken in a pretty pathetic way ... >I perceive the "basic problem" as the fact that sampling variance is >(relatively) easily calculated for a SRS, while it is more difficult >to calculate under almost _any_ other type of sampling. sure ... but, books ONLY seem to discuss the easy way ... and i do too ... because it seems rather straight forward ... but, given time constraints ... it never goes further than that ... > Whether it is enough more difficult that one would REALLY like to avoid >it in an elementary course is a judgement call; but for the less >quantitatively-oriented students with whom many of us have to deal, we >_would_ often like to avoid those complications. Certainly dealing with >the completely _general_ case is beyond the scope of a first course, so >it's just a matter of deciding how many, and which, specific types of >cases one is willing to shoehorn into the semester (and what "previews >of coming attractions" one wishes to allude to in higher-level courses). however, we do become sticklers for details ... and force students to use the correct CVs, make the right CIs, ... do the t tests correctly ... and heaven forbib if you get off a line or two when reading off the values from the t table ... >Seems to me the most sensible "adjustment" (and of a type we make at >least implicitly in a lot of other areas too) is > = to acknowledge that the calculations for SRS are presented >(a) for a somewhat unrealistic "ideal" kind of case, i would stress ... really unrealistic ... >(b) to give the neophyte _some_ experience in playing this game, and then leave them hanging >Some textbooks I have used (cf. Moore, "Statistics: Concepts & >Controversies" (4th ed.), Table 1.1, page 40) present a table giving the >margin of error for the Gallup poll sampling procedure, as a function of >population percentage and sample size. Such a table permits one to show >how Gallup's precision varies from what one would calculate for a SRS, >thus providing some small emphasis for the cautionary tale one wishes to >convey. but ... in moore and mccabe ... the stress throughout the book ... is on SRSes ... and no real mention is made nor solutions to ... the problems that it will be a rare day in analysis land ... for the typical person working with data ... to be doing SRS sampling ... it's just not going to happen the bottom line, IMHO, is that we glide over this like it is not a problem at all ... when we know it is > > Donald F. Burrill [EMAIL PROTECTED] > 184 Nashua Road, Bedford, NH 03110 603-471-7128 _ dennis roberts, educational psychology, penn state university 208 cedar, AC 8148632401, mailto:[EMAIL PROTECTED] http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: SRSes
Hi, Dennis! Yes, as you point out, most elementary textbooks treat only SRS types of samples. But while (as you also point out) some more realistic sampling methods entail larger sampling variance than SRS, some of them have _smaller_ variance -- notably, stratified designs when the strata differ between themselves on the quantity being measured. On Tue, 24 Jul 2001, Dennis Roberts wrote: > most books talk about inferential statistics ... particularly those > where you take a sample ... find some statistic ... estimate some error > term ... then build a CI or test some null hypothesis ... > > error in these cases is always assumed to be based on taking AT LEAST a > simple random sample ... or SRS as some books like to say ... > > but, we KNOW that most samples are drawn in a way that is WORSE than SRS I don't think _I_ know this. I know that SOME samples are so drawn; but (see above) I also know that SOME samples are drawn in a way that is BETTER than SRS (where I assume by "worse" you meant "with larger sampling variance", so by "better" I mean "with smaller sampling variance"). > thus, essentially every CI ... is too narrow ... or, every test > statistic ... t or F or whatever ... has a p value that is too LOW > > what adjustment do we make for this basic problem? I perceive the "basic problem" as the fact that sampling variance is (relatively) easily calculated for a SRS, while it is more difficult to calculate under almost _any_ other type of sampling. Whether it is enough more difficult that one would REALLY like to avoid it in an elementary course is a judgement call; but for the less quantitatively-oriented students with whom many of us have to deal, we _would_ often like to avoid those complications. Certainly dealing with the completely _general_ case is beyond the scope of a first course, so it's just a matter of deciding how many, and which, specific types of cases one is willing to shoehorn into the semester (and what "previews of coming attractions" one wishes to allude to in higher-level courses). Seems to me the most sensible "adjustment" (and of a type we make at least implicitly in a lot of other areas too) is = to acknowledge that the calculations for SRS are presented (a) for a somewhat unrealistic "ideal" kind of case, (b) to give the neophyte _some_ experience in playing this game, (c) to see how the variance depends (apart from the sampling scheme) on the sample size (and on the estimated value, if one is estimating proportions or percentages), (d) in despite of the fact that most real sampling is carried out under distinctly non-SRS conditions, and therefore entails variances for which SRS calculations may be quite awry; and = to have yet another situation for which one can point out that for actually DOING anything like this one should first consult a competent statistician (or, perhaps, _become_ one!). Some textbooks I have used (cf. Moore, "Statistics: Concepts & Controversies" (4th ed.), Table 1.1, page 40) present a table giving the margin of error for the Gallup poll sampling procedure, as a function of population percentage and sample size. Such a table permits one to show how Gallup's precision varies from what one would calculate for a SRS, thus providing some small emphasis for the cautionary tale one wishes to convey. Donald F. Burrill [EMAIL PROTECTED] 184 Nashua Road, Bedford, NH 03110 603-471-7128 = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
SRSes
most books talk about inferential statistics ... particularly those where you take a sample ... find some statistic ... estimate some error term ... then build a CI or test some null hypothesis ... error in these cases is always assumed to be based on taking AT LEAST a simple random sample ... or SRS as some books like to say ... but, we KNOW that most samples are drawn in a way that is WORSE than SRS ... thus, essentially every CI ... is too narrow ... or, every test statistic ... t or F or whatever ... has a p value that is too LOW ... what adjustment do we make for this basic problem? _ dennis roberts, educational psychology, penn state university 208 cedar, AC 8148632401, mailto:[EMAIL PROTECTED] http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
