> If you want, you can send me your presentation some time, and I can > give you some feedback and/or we can do some brainstorming about it.
I now have a draft presentation: http://www.csse.uwa.edu.au/~john/drafts/LCA_Da11-diag.pdf On Thu, Dec 30, 2010 at 9:27 PM, Vincent van Ravesteijn <v...@lyx.org> wrote: >> its rather psychological than technical issue that if keytest generate 5 bugs >> per week you easily break through certain threshold after which motivation >> to fix remaining bugs decrease. i'm sure Vincent could give you better >> insights ;) > > From a technical point of view it's very good to be able to > stress-test the software. But some bugs found by keytest require too > much effort in comparison to the improved usability of LyX. Then I'd > rather don't want to know about them (it's not like we're running a > nuclear plant here). Would you like to suggest a solution? Perhaps artificially rate-limiting the bug reports not relating to recent regressions to say 1/week? It seems that the majority of bugs found by keytest would eventually be hit by users. I understand that even the threaded export bugs have started hitting users. It seems that the flood of bugs found when the keytest is first turned on could be a large part of psychological issues. Also if we ratelimit the bugs, the bugs reported could be chosen on the basis of the effect on usability. I was wondering if Keytest also has the potential to provide psychological advantages, e.g. 1) Avoid floods of bug reports in the beta cycle by moving them earlier 2) Always feel ready to ship, and ready to test. Frequently used functions have shortcuts, and so breakage in those functions should be found by keytest quite quickly. This should mean that one could send a friend an arbitrary snapshot and be somewhat confident that they notice how cool your latest feature is before they hit any embarrassing bugs. One of my hopes is that the trunk could feel stable, leading to more power users thinking "I know how to deal with regressions and trunk feels pretty stable, lets try doing real work with it". Since bugs that occur when doing real work are the most valuable these would help keep LyX ready to ship. One hope would be that this could avoid the risk that it would feel that the next version will never be as reliable as the last. > As a final remark, I think that if you want to have rock-solid > software, mon-keytesting will not suffice. I bet I can find more > critical bugs than the keytesting will. For instance by entering > commands in the command buffer. I'm sure your monkeytest will type > some valid lfuns in there, but it will take you some ages of the > universe before it enters an lfun (or a combination of lfuns) that > will crash LyX. There is a paper suggesting that only roughly %20 of bugs will be found by monkeytesting, but the most serious bugs are more likely to be detected. Crashing every time a document containing a math-inset is likely to be found quickly. An function only accessible by LFUN is less likely to vital to many users. For rock-solid software I suspect you'd want multiple ways of finding bugs; since Monkeytesting is cheap it should probably be included. One could formally prove that an embedded system is correct, and yet testing could still discover that the CPU catches fire under heavy load. My personal motivation is more in finding bugs that cause major breakage quickly, and avoiding the big dip in quality that typically occurs between opening trunk and releasing a beta. > In that sense I'm not sure whether a technique like this has a future. > If I would invent something I would go one level deeper and would call > the functions of a class with random parameters. For example, if you > have the function Math::sqrt(x), it will not take long until one finds > out that a negative value of x will be a problem, so the algorithm > automatically finds that there should be a condition that x can never > be smaller than 0. This can be used in the functions that call this > sqrt-functions and these assert when x becomes smaller than 0 and so > forth and so forth. I thought I would briefly mention static analysis. Static analysis has the potential to show exactly what the code defect is, making fixing the code defect trivial. Static Analysis tools are difficult to develop though. > In the end, what would be the best way to test software ? What > available software is there to test it ? Is your monkeytesting filling > a niche in there ? What advantages does the monkeytesting have over > other approaches ? I see the main advantage as being that Monkeytesting has the potential to be essentially free. E.g. If 1% of Ubuntu users were willing to run keytest in the background every OSS project could be monkeytested many times over. So regardless of what other testing is also done, monkey testing could be done as well if it provides any advantage. > See it as stresstesting (or monkeytesting) the logic in your > presentation ;). > > Greetings, > > Vincent Thank you for your feedback :) -- John C. McCabe-Dansted
