Just came across this article. It seems pertinent to this discussion. http://www.sciencebasedmedicine.org/psychology-journal-bans-significance-testing/
On Wed, Feb 25, 2015 at 10:55 AM, Ian Clark <[email protected]> wrote: > On Wed, Feb 25, 2015 at 3:31 PM, Devon McCormick <[email protected]> > wrote: > > Ian - setting up a framework for this sort of experiment could prove > > valuable in a wide variety of fields, not just audio testing. > > > > However, you may be overshadowing the point of the example from the book: > > it's an illustration of the relevance of prior knowledge. As I remember > > it, not having the book in front of me, it goes something like this: > > > > There are these three examples of evidence supporting a hypothesis: > > > > 1) A lady claims to be able to distinguish, by tasting a cup of tea with > > milk, whether the tea was added before the milk or the milk before the > > tea. You test her ten times and she is correct every time. > > > > 2) Someone claims to be able to distinguish by ear a score written by > > Mozart from one not written by Mozart. You test him ten times and he is > > correct every time. > > > > 3) A drunken friend claims to be able to predict the result of a coin > > toss. You test him ten times and he is correct every time. > > > > Since the empirical evidence in all three cases is identical, why would > we > > not believe all three hypotheses to be equally well-proved? > > Aha! -- do I detect a Bayesian? Or a disciple of de Finetti? > > I recall a Real statistician taking me aside and telling me: "the > trouble with you people is that you don't think anything's happened in > Statistics in the last 20 years." -- and that was way back in 1989. > > I wouldn't know how to write a program based on a Bayesian analysis to > replace my classical "H0-rejection" analysis. I suspect it's no use > even trying. De Finetti, I recall, required you to input your a-priori > subjective prediction of the outcome as a normalized distribution > before anything else. > > Maybe I'll just stick to a disclaimer… > "This assessment is based on the assumption that you are not an > occultist, a superhero, mentally certifiable or under the influence of > drugs or alcohol." > …so bang goes all my user audience of musicians and songwriters! > :-D > > > > On Wed, Feb 25, 2015 at 3:31 PM, Devon McCormick <[email protected]> > wrote: > > Ian - setting up a framework for this sort of experiment could prove > > valuable in a wide variety of fields, not just audio testing. > > > > However, you may be overshadowing the point of the example from the book: > > it's an illustration of the relevance of prior knowledge. As I remember > > it, not having the book in front of me, it goes something like this: > > > > There are these three examples of evidence supporting a hypothesis: > > > > 1) A lady claims to be able to distinguish, by tasting a cup of tea with > > milk, whether the tea was added before the milk or the milk before the > > tea. You test her ten times and she is correct every time. > > > > 2) Someone claims to be able to distinguish by ear a score written by > > Mozart from one not written by Mozart. You test him ten times and he is > > correct every time. > > > > 3) A drunken friend claims to be able to predict the result of a coin > > toss. You test him ten times and he is correct every time. > > > > Since the empirical evidence in all three cases is identical, why would > we > > not believe all three hypotheses to be equally well-proved? > > > > > > On Wed, Feb 25, 2015 at 10:19 AM, Ian Clark <[email protected]> > wrote: > > > >> @Chris - just what I've been looking for! > >> > >> However did I miss it? -- I guess it must have eluded my choice of > >> keywords. > >> Next time I'll use Google with site:jsoftware.com > >> > >> On Wed, Feb 25, 2015 at 2:27 PM, chris burke <[email protected]> > wrote: > >> >> Has anyone built a standalone Mac app using JQt? > >> > > >> > Please see http://www.jsoftware.com/jwiki/Guides/J8%20Standalone > >> > > >> > On 25 February 2015 at 06:09, Ian Clark <[email protected]> > wrote: > >> > > >> >> @Henry -- thanks for your comments. Great! > >> >> > >> >> IMO this is just the sort of discussion I would like to see aired in > >> >> public. Though maybe do the more philosophical stuff in Chat? > >> >> Ideally I would like a summary of the J community's findings > >> >> documented on a Jwiki page for wider consumption. > >> >> > >> >> Further comments in-line… > >> >> > >> >> > >> >> On Wed, Feb 25, 2015 at 4:27 AM, Henry Rich <[email protected]> > >> wrote: > >> >> > We should take this off-group, but I'm replying in public because > if > >> I'm > >> >> > wrong I would like to be corrected (and I'm only an amateur > >> >> statistician): > >> >> > >> >> That's exactly why I'm appealing to the forum too. > >> >> …To the annoyance of Real Statisticians, no doubt, because this must > >> >> be elementary stuff to them. > >> >> But Wikipedia -- which you'd expect to give simple answers to simple > >> >> questions which laypeople want to ask and need to ask -- approaches > >> >> the whole issue like a cat circling a bowl of hot porridge. > >> >> …If you're a layperson, just try working out how to score the "Lady > >> >> Tasting Tea" experiment from these pages… > >> >> https://en.wikipedia.org/wiki/Binomial_test > >> >> https://en.wikipedia.org/wiki/Binomial_distribution > >> >> https://en.wikipedia.org/wiki/Bernoulli_distribution > >> >> https://en.wikipedia.org/wiki/Bernoulli_trial > >> >> https://en.wikipedia.org/wiki/Bernoulli_process > >> >> > >> >> As a Human Factors *engineer* -- I've been a professional *user* of > >> >> hypothesis-testing but an amateur Statistician. > >> >> …Or should that be Probabilist? Or even Epistemologist? > >> >> > >> >> Plus… now I'm retired, I'm getting rusty. > >> >> > >> >> Plus… I can't find precise enough documentation of JAL verb: > >> binomialprob. > >> >> Like… what's the semantics of the 3rd entry of (y) (styled "minimum > >> >> number of successes (s)") when y has only 3 entries? Can it be called > >> >> "minimum" any more? What I've concluded, after a bit of RTFC plus a > >> >> few idiot tests, is: > >> >> > >> >> (binomialprob 0.5,N,s) -: (binomialprob 0.5,N,s,N) > >> >> > >> >> Plus… has this doggie got 2 tails or just 1?? > >> >> > >> >> > >> >> > I think you are calling binomialprob correctly but I have some > >> >> objections to > >> >> > your use of the result. > >> >> > > >> >> > 1. I think your rejectH0 should use 1 - -: CONFIDENCE instead of > >> >> > 1-CONFIDENCE. > >> >> > > >> >> > The question is, "How likely is a result as weird as I am seeing, > >> >> assuming > >> >> > H0?" You should not bias "weird" by assuming that weird results > will > >> be > >> >> > correct guesses - they could just as likely be incorrect guesses. > To > >> >> ensure > >> >> > that you reject 95% of the purely-chance deviations of a certain > size, > >> >> that > >> >> > 95% should be centered around the mean, not loaded toward one side. > >> >> > >> >> The "1-tail-or-2?" question -- or so I thought at first. > >> >> But it's deeper than that. It's much more serious. Serious enough to > >> >> be the key issue for me. > >> >> Which is precisely why I want to be *sure*. Sure enough to argue my > >> >> case to a determined layperson. Not merely make an inspired guess, as > >> >> most people would in an industrial situation (…knowing no one else > >> >> knows enough statistics to dare to challenge you!) > >> >> > >> >> What I understand @Henry to be saying is: should the 5% area under > the > >> >> binomial distribution curve, which sets the pass/fail threshold, be > >> >> shared equally between both tails? Even if one tail happens to be in > >> >> fairyland? > >> >> > >> >> What I mean by that last remark is… > >> >> If The Lady Tasting Tea (TLTT) gets every trial *wrong*, then she's > >> >> *not* a monkey flipping a fair coin. It's a very biased coin! > >> >> She is sending a strong signal that she can be depended upon (…with > X% > >> >> confidence) to make the wrong decision. > >> >> But I don't want to credit her this as evidence to support her claim > >> >> she can tell the difference (…at least, not tell it correctly). > >> >> This is what makes TLTT different from detecting a biased coin by > >> >> repeated tosses. > >> >> > >> >> What's to do? > >> >> > >> >> > >> >> > are there really people who think optical might be better than > USB?? > >> >> > >> >> Oh-ho-ho! -- yes, they can still be found. > >> >> Hi-Fi buffs have not become extinct, and the (undead?) audio industry > >> >> still lives off their lifeblood. > >> >> > >> >> > >> >> > This is digital communication, no? 44K samples/sec, 2 channels, 20 > >> >> bits/sample, > >> >> > needs 2Mb/sec max out of 480Mb/sec rated USB speed... how could > that > >> not > >> >> be > >> >> > enough? > >> >> > >> >> My interlocutor claims it's like the group was there, in his front > >> >> room, playing "just for him". > >> >> Now this guy is an intelligent chap, a developer of digital musical > >> >> instruments and a sound engineer as well as being an accomplished > >> >> musician. He sends me two MP3s (…yes, lossy MP3s!) to support his > >> >> claim. I drop these into Audacity and inspect the waveform at very > >> >> fine detail and I cannot for the life of me detect any difference. > >> >> So I know, as sure as God made little Apples, that I'm not going to > >> >> *hear* any difference. > >> >> But I've got lo-fi ears. In fact I'm half-deaf. Most of what I hear I > >> >> imagine. Mostly I get it right with people (I think…) But I don't > know > >> >> what subliminal cues I'm using to do so. It's the "clever Hans" > >> >> effect. > >> >> > >> >> Maybe there are people who *can* tell the difference? But from my > >> >> pondering the figures, like you have, plus eyeballing the waveforms, > >> >> we're talking about magical superpowers here. > >> >> > >> >> > >> >> > It was ever thus... when I last looked at this sort of thing, 20 > years > >> >> back, > >> >> > the debate was whether big fat expensive cables would make a > >> difference. > >> >> > Bob Pease, a respected analog engineer, pointed out that it was > >> >> impossible, > >> >> > and James Randi had a bet that no one could discern $7000 cables > from > >> >> > ordinary speaker wire, but still the non-EEs have their > >> superstitions...] > >> >> > >> >> That's around the time my son was spending all his pocket-money on > big > >> >> fat speaker cables and gold-plated jack-plugs. > >> >> Now he's teaching a Theory of Knowledge course (…yes, Epistemology!) > >> >> at a school in Hong Kong. He is greedy to get his hands on my little > >> >> program, and dispel a few lingering superstitions masquerading as > >> >> received wisdom about science. > >> >> > >> >> I want to package it up and send it to him, but I don't want to ask > >> >> him to install J on his Mac because not only will he grouse like heck > >> >> about fairy software but it will discourage him sharing the app with > >> >> his colleages, who share his sentiments. > >> >> > >> >> I know how to package up a standalone Mac app in J602, but J602 and > my > >> >> packaged apps no longer work out-of-the-box on the Mac under Yosemite > >> >> (it's to do with 32-bit Java). Has anyone built a standalone Mac app > >> >> using JQt? If so I'd dearly love to see a monkey-see monkey-do page > on > >> >> Jwiki. I'll write one myself, but it'll be a year before I can get > >> >> round to it. > >> >> > >> >> > >> >> > 2. Why 95%? I would fear that someone who is thinking about > optical > >> >> cable > >> >> > would rest uneasy with a 5-10% chance that they have not spent > enough > >> on > >> >> > quality audio. Why not simply report, "A monkey with a coin to > toss > >> >> would > >> >> > do as well as you y% of the time. Most researchers accept results > as > >> >> > significant only if the monkey would do as well less than 5% of the > >> time. > >> >> > Take more samples if you want less uncertainty." > >> >> > >> >> 95% is just for the sake of argument. 99% is there as an option. IMO > >> >> more options are neither necessary nor advisable. > >> >> The number of trials can be varied too. I'd like to offer 10 or 20 > >> >> trials. But 20 gets tedious, so I'm offering the option to give up > >> >> when you're bored and score the number you've done. > >> >> (This is an app for discretionary users -- we're not paying our > >> >> subjects $10 a session.) > >> >> > >> >> Anything under 7 trials fails to reject H0 however many successes. > But > >> >> that's dependent on the value of CONFIDENCE and how it's to be > >> >> applied. But only to make a difference of 1 or maybe 2 trials. > >> >> I'm finding in practice that with such a low number of trials as 10, > >> >> anything short of 100% correct is statistical hairsplitting when it > >> >> comes to rejecting H0. With 20 trials there's more leeway: you're > >> >> allowed to get 3 or 4 wrong before the app rubbishes you. > >> >> > >> >> As for your wording: it's theoretically sound, but a trifle > insulting. > >> >> Performing musicians have sizeable egos and wouldn't like to be rated > >> >> along with performing monkeys. :-) > >> >> > >> >> Ian > >> >> > ---------------------------------------------------------------------- > >> >> For information about J forums see > http://www.jsoftware.com/forums.htm > >> >> > >> > ---------------------------------------------------------------------- > >> > For information about J forums see > http://www.jsoftware.com/forums.htm > >> ---------------------------------------------------------------------- > >> For information about J forums see http://www.jsoftware.com/forums.htm > >> > > > > > > > > -- > > Devon McCormick, CFA > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
