On Wed, Apr 19, 2023 at 10:28 PM Rick Byers <rby...@chromium.org> wrote:
> On Wed, Apr 19, 2023 at 3:41 PM Philip Jägenstedt <foo...@chromium.org> > wrote: > >> I wonder if we can get enough confidence with less work than >> investigating 40 randomly chosen sites from UseCounter hits. >> >> This is a population proportion problem, and >> https://sample-size.net/confidence-interval-proportion/ is a useful >> tool. If you check 40 cases and find no breakage (N=40, x=0) that gives us >> 95% confidence that breakage is less than 7.2% of samples in this data set. >> If it's useful to check that much depends on the value of the use counter. >> >> Is https://chromestatus.com/metrics/feature/timeline/popularity/4463 the >> right use counter, and has it reached stable yet? Why is marked as obsolete? >> >> For purposes of illustration, let's use 0.04% from earlier in the thread >> and say we want to be (95%) confident that real breakage is less than >> 0.01%. Then we just need to get below 25% in the linked tool, and checking >> 11 samples and finding nothing is enough to do this. >> > > I like your more sophisticated math, but it's <0.001% that we want I'm > afraid, not 0.01%. So, if I'm following your instructions right, that's ~42 > samples to have 95% confidence :-) > > Of course 0.001% is just a rough guideline that has often proven to be too > high or too low. So this is all a judgement call anyway. > Ah, then I understand where the number 40 came from (0.04/0.001). I may well be wrong, but I think to get a 40-fold reduction in the upper bound, we need to sample until our upper bound is <=0.025, and with CL=90 (to get a 95% one-sided CI) we'd need 119 samples with no breakage to get there. But definitely let's not do that, I was probing to see if we could do less work, not more. Checking 40 cases and finding nothing is better than we've done in many cases, so the risk feels quite low. -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYeZciSiwOy9321jtr4jgL-cBjfUPjqqA3mpA2Bpb2V%2BZA%40mail.gmail.com.
