On Sat, Oct 07, 2017 at 11:33:33AM -0400, Jeffrey Walton wrote: > On Sat, Oct 7, 2017 at 11:25 AM, Salz, Rich <rs...@akamai.com> wrote: > > > > > >> I suggest we not have this debate now. We'll have a lot more data towards > >> the end of the month and we can have an informed discussion then. > > > > > That is what I am asking for. More information so that the entire WG can > > make an informed decision. And I was only laying out an option that does > > not seem to have been considered before. > > The group (or the IETF) might also consider a policy to answer Ilari > Liusvaara's question, "What you think is acceptable failure rate?" > > That is a governance issue. It should probably be [nearly] written in > stone and applied equally to all problems and decisions. >
Unfortunately, things are actually more complicated than that. I suspect that none of the figures "minimal", 1.5% nor 3.4% are actually accurate, due to "survivorship bias" (survive to be tested). This is both due to large variance in results, and Google and FB disagreeing on impact of the record type hack. I think Attributing the differences to survivorship bias (or other similar statistical bias) makes much more sense than attributing the differences to random chance (I presume the sample sizes are large enough to easily resolve even 0.1% differences) or a testing mistake. If asked to guess which result is the closest to the true value, I would guesss Google's (which is also the largest value). But I do not have any idea even which direction the true value is (often in studies it is rather easy to guess the direction of the true value, even if one can not guess the correction magnitude). The nasty issue in testing is that there are several classes of connections, with quite widely varying properties: 1) Residential wired 2) Wireless Mobile 3) Enterprise (includes some schools) 4) Satellite (most probably of minimal use). I would expect that with residential wired, the failure rates are minimal, whereas with Enterprise, the failure rates would be pretty substantial. With wireless mobile in the middle. I have no idea how satellite would stack up. I suspect the main factor in differences was proportion of enterprise networks that were tested. Which would imply that the enterprise failure rate is much higher than 3.4%. And even getting the failure rate reduced to 1%, the enterprise failure rate would still be substantially higher than 1%. And analyzing individual middleboxes to determine the kind of intolerance is only useful for guiding what kind of modifications to test on the field, since estimating any useful statistics from just the devices or software behavior is virtually impossible. -Ilari _______________________________________________ TLS mailing list TLS@ietf.org https://www.ietf.org/mailman/listinfo/tls
