Paying for what is required
At 11:09 PM -0800 1/26/02, Kyle Lussier wrote: I seem to be getting two conflicting viewpoints: #1 Vendors can only be trusted to be interoperable on their own, and can not be forced to conform. #2 Vendors absolutely can't be trusted to be interoperable, without conformance testing. Missing is #3 Vendors can't be trusted to be interoperable without interoperability testing. VPNC performs conformance testing for IPsec, and there are plenty of examples of our members who conform but do not interoperate without a lot of knob twiddling and an occasional bug fix. The long experience with IPsec interoperability events (which VPNC does not hold) has shown that A, B, and C might conform, and A and B can interoperate fine, but A and C cannot interoperate. This is usually due to administrative interfaces either not having the right knobs, the defaults for C being valid for conformance but not for interoperability, or weird magic. Is interoperability testing needed for end users? Possibly, but it won't happen until someone comes up with a good business model for the testing agency. When anyone comes up with one, I'd love to hear it. VPNC was originally formed to do good interop testing for the IPsec industry, but when we figured out what that would cost all of the members, there was no longer any interest. The basic problem: either each of the 35 members is responsible for running and debugging the test with the other 34 members, or they are willing to pay someone to run and debug the 1225 (35^2) tests for them. In the former case, the best statement of why that was not attractive was if I have a staff person who has that much skill with our product and the at least 100 hours it will take, I have much more important work for them. In the latter case, there was immediate history of another interop testing agency who both charged a large amount of money to do the tests and a fair amount of vendor staff time to do debugging in order to do about one fifth the number of tests. Thus, VPNC is left doing conformance testing with verifiable results, which is admittedly not nearly as valuable to end users. (See http://www.vpnc.org/conformance.html, particularly near the end, for more details.) VPNC also does some small-to-medium sized interop demos, but these are not formal interop tests with formal results. Of course, this is not to say that formal interoperability testing is impossible. There are examples of where it happens today in the Internet industry. But there are probably one or two orders of magnitude of examples of where it does not happen. Informal interoperabilty events, where there are no results published but lots of good interaction between vendors, have helped the industry a great deal but are largely invisible to end users (and still don't produce the level of interoperability that people in this discussion say is required). Does the IETF or ISOC want to get into either conformance or interoperability testing? It is fairly safe to say this not going to happen without a business model to pay for the short-term and long-term costs. So far, no business model has appeared in the discussion. --Paul Hoffman, Director --VPN Consortium
Re: Fwd: Re: IP: Microsoft breaks Mime specification
I fear that I made a notable mistake in the original suggestion that we somehow deal with Conformance. In fact, I agree entirely that the issue of concern is Interoperability. As I have noted before, I also agree that the IETF is the wrong place to deal with the problem by serving as the judge and jury. Whatever might be done needs to be a Market Based Initiative. My apologies for setting the wrong course...\Stef At 17:33 -0800 26/01/02, Bob Braden wrote: * * But the use of a trademark, which stands for complies with RFCs * could be incredibly valuable. * Kyle, I suggest that you read RFCs 1122 and 1123 from cover to cover, and then ponder whether the nice-sounding phrase complies with the RFCs has any useful meaning. Perhaps you will begin to understand why the IETF Way is interoperability testing, not conformance testing But you are free to make your proposal at IAB plenary of the next IETF. This discussion is in a loop. Bob Braden
Why does Valdis trust UL?
Title: Why does Valdis trust UL? At 10:19 PM -0500 1/26/02, [EMAIL PROTECTED] wrote: I have in my bedroom a night light, which I purchased at a local grocery store. It has a UL logo on it, which doesn't tell me much about its suitability as a night light (I can't tell if it's bright enough, or if it's too bright, or what its power consumption is), but it *does* tell me 2 things: 1) It has been *tested* and found free of any known safety design problems. It may not *work* as a night light, but it won't shock me when I go to throw it in the trash can because it's not suitable. 2) A high enough percentage of night light manufacturers get UL listed that I can afford to be suspicious of any company that doesn't have the logo on their product. Underwriters Laboratories, Inc. is a non-profit corporation that was founded in 1894. This article describes the process UL uses for developing their standards. Many UL standards receive ANSI certification. According to the article, UL relies on information from a number of sources while developing a standard. UL tests products submitted by its customers for *conformance* to its standards. UL's reputation depends on the rigor and independence of their testing. I don't know how it costs to submit a product for testing, but obtaining UL certification isn't free. UL's certification program is successful, because when consumers like Valdis (and me) see a UL label, they believe in its value. As Valdis points out, the value of the label has limits. Certification isn't the work of a volunteer organization like the IETF. It could be the work of an organization like Underwriters Labs. This would be a good thing for Internet standards, imho. One idea proposed multiple times in this meandering discussion is that those advocating testing should put up or shut up -- create a testing organization or move on to other topics. I concur with both those suggestions. I'm sure you'll all be pleased this is my last word on the topic. best, -- john noerenberg [EMAIL PROTECTED] -- While the belief we have found the Answer can separate us and make us forget our humanity, it is the seeking that continues to bring us together, the makes and keeps us human. -- Daniel J. Boorstin, The Seekers, 1998 --
