On Sat, Mar 18, 2006 at 09:31:27PM +0100, Roland Mainz wrote: > Ouch... every single binary with all possible combinations of input ? > That's almost impossible for a single person .. ;-(
No one said you have to manually test every possible input to every possible program. In many cases you may be able to use semi-automated means to rule out a need for any manual testing at all (for example, if you can show that every reference to a literal is through a const char *, or if a program contains no string literals at all). You also don't have to do it yourself. Perhaps you can enlist 50 like-minded people, each of whom can test one or two commands a day. At that rate you'd have usr/src/cmd cleaned up in just a few weeks. But, yes, you do have to test each program at least a little bit using test cases designed to trigger the kind of problems that you expect you may find, or show why it's unnecessary. > The only thing which I could provide myself is something like a stacked > build test, e.g. build ON, boot it, build ON inside it and then boot > that version and built ON in it, too. And then compare builds 2 and 3 > for differences and/or build+boot failures. That's grossly inadequate. For example, one of the problems found was in ikeadm, which would never be exercised during any part of a build/BFU cycle. > Alternatively I could just post the patches (including a deadline when > the patch should be commited (AFAIK similar to the Sun concept of a > "flag day")) and let the community test it for breakage. A flag day just means someone has made a change which requires some action on your part to get past it and continue building/using/testing the contents of the gate. For example, you might need to BFU to get past a kernel/library interface change, or you might need to upgrade some tools on your build machine. Flag days have absolutely nothing to do with the idea that a potentially broken change somehow becomes less broken as it ages. Posting diffs for review is a good idea. Posting diffs to encourage others to test your changes and report problems you might not find in your own testing is also a good idea. But posting diffs as a substitute for doing any kind of meaningful testing of your own is unacceptable. Posting something to a mailing list and waiting N days for someone to tell you it's broken is not a form of testing; it's a broken development methodology that attempts to offload your personal, individual responsibility for your change onto an arbitrarily large number of unaccountable third parties. Not only isn't this allowed for testing, it's not allowed for code review, either. The ON CRT notes this at http://opensolaris.org/os/community/onnv/crt/rti-nits/: "The list of code reviewers should be individuals who have actually reviewed the code, not an alias for a group of people who were given the opportunity to do so." In fact your approach sounds a lot like a standard management strategy: "This project has to integrate by $DEADLINE, no matter how broken it is, so you'd (note offloading of responsibility here) better work harder to make sure it's ready!" We have a pretty dim view of that school of software engineering. > There is a catch... the public query interface for the bug database > isn't very sophisticated (well, some people here even consider it being No, it isn't very good at all. But a search for 'gcc' or 'constant' or 'string literal' should at least be somewhat useful. > a bad joke compared to the bugzilla query interface) and without having > access to the patches in the database which shows what was done to fix > the particular issue it's even harder (AFAIK there was another thread > about patch access in the bug database this month) if not impossible You can query for the bug ID in the history field in OpenGrok. If the fix was putback after June 14, 2005 (true for most of these), you'll be able to see it. For example, a search for 'writes into string constants' brings up the ikeadm example I referenced, which could also be found by searching for 6281951. I'll be the first to admit that b.o.o is terrible, but nobody has perfect tools and you can always find some excuse for not doing any work. If you want to make this change, I'll sponsor it for you, but you - and people you recruit - are going to have to do the legwork. If you don't think it's worth the time it would take, you can console yourself with the knowledge that your conclusion is the same one that dozens of engineers before you have reached. -- Keith M Wesolowski "Sir, we're surrounded!" Solaris Kernel Team "Excellent; we can attack in any direction!" _______________________________________________ opensolaris-discuss mailing list opensolaris-discuss@opensolaris.org
