Henry, Henry, I've ALWAYS LOVED your replies, and this one in particular. I have to send you this note even though I have not responded to J-forum or you in 10+ years, nor touched J. I think my PC in the closet has J 4 on it. (don't even rem how the version #s appear). Dick Penny
----- Original Message ----- From: "Henry Rich" <[email protected]> To: [email protected] Sent: Tuesday, February 24, 2015 8:27:51 PM Subject: Re: [Jprogramming] Tea-tasting for non-statisticians We should take this off-group, but I'm replying in public because if I'm wrong I would like to be corrected (and I'm only an amateur statistician): I think you are calling binomialprob correctly but I have some objections to your use of the result. 1. I think your rejectH0 should use 1 - -: CONFIDENCE instead of 1-CONFIDENCE. The question is, "How likely is a result as weird as I am seeing, assuming H0?" You should not bias "weird" by assuming that weird results will be correct guesses - they could just as likely be incorrect guesses. To ensure that you reject 95% of the purely-chance deviations of a certain size, that 95% should be centered around the mean, not loaded toward one side. [are there really people who think optical might be better than USB?? This is digital communication, no? 44K samples/sec, 2 channels, 20 bits/sample, needs 2Mb/sec max out of 480Mb/sec rated USB speed... how could that not be enough? It was ever thus... when I last looked at this sort of thing, 20 years back, the debate was whether big fat expensive cables would make a difference. Bob Pease, a respected analog engineer, pointed out that it was impossible, and James Randi had a bet that no one could discern $7000 cables from ordinary speaker wire, but still the non-EEs have their superstitions...] 2. Why 95%? I would fear that someone who is thinking about optical cable would rest uneasy with a 5-10% chance that they have not spent enough on quality audio. Why not simply report, "A monkey with a coin to toss would do as well as you y% of the time. Most researchers accept results as significant only if the monkey would do as well less than 5% of the time. Take more samples if you want less uncertainty." Henry Rich On 2/24/2015 9:46 PM, Ian Clark wrote: > Addon 'stats/base/distribution' defines the verb: binomialprob. > Am I using it correctly? > Please cast a beady eye over my train of thought, as I've set it out below... > > I've written an app in J to administer a double-blind test which > reruns the classic experiment described by David Salsburg in "The Lady > Tasting Tea". But in place of pre- and post-lactary tea, I play a > snatch of one of two soundfiles in a series of 10 trials to ascertain > if she really can tell the difference. > >>From her answers I compute 2 numbers: > N=: number of trials (typically 10 or 20) > s=: number of successes. > I have also set up an adjustable parameter: > CONFIDENCE=: 0.95 NB. (the 95% confidence limit) > > Instead of using binomialprob directly, I define 2 verbs: > pH0=: 4 : 'binomialprob 0.5,x,y' > rejectH0=: 4 : '(1-CONFIDENCE) > x pH0 y' > > (p=. N pH0 s) is the likelihood of s arising at random under the > "null hypothesis" (H0), viz that she's just guessing with > probability=0.5 of success. > (N rejectH0 s) returns 1 iff p is too low, as determined by > CONFIDENCE, implying the null hypothesis (H0) can safely be rejected. > This Boolean value triggers one of 2 messages: > 1 --> You can tell the difference. > 0 --> You're just guessing. > > That's straining the epistemology, I know. But I don't expect a > non-statistician to make much sense of a statistically kosher message, > such as: > 0 --> This program has decided that the (null) hypothesis that your > results have arisen by pure guesswork cannot be safely rejected on the > evidence alone of these 10 trials. > > There's gratifying interest in the music/audiophile community in such > a sound-test, if it's packaged up and made easy-to-use. Questions > like: "can you hear any improvement if you use an optical cable > instead of a USB one?" come up all the time. And, as I've discovered > for myself, even a strong impression of improvement may not stand up > to this sort of scrutiny. It's the placebo effect. > > So not only do I need to assure the soundness of the statistical > theory and its J implementation, plus my use of it, but also publish a > proper write-up (in Jwiki) which makes sense to a non-statistician. > This after all is *the* foundation experiment in the history of > Statistics. > But AFAICS its treatment in Wikipedia leaves much to be desired. > See for instance: https://en.wikipedia.org/wiki/Binomial_test > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
