On 02/25/2013 03:45 PM, Amos Jeffries wrote: > On 26/02/2013 11:06 a.m., Alex Rousskov wrote: >> The attached patch adds about 700'000 test cases for eight >> SBuf::find() methods. Needless to say, they are generated rather than >> manually written. They take a couple of seconds to run. Configuration >> options allow us to increase or decrease the number of test cases if >> needed.
> Why do we need so many tests? I do not know how many tests are needed. Somebody can study that (if they feel that reducing the number of generated tests is important). More on that below. > Surely a selection of lengths from the middle, and specific scans of all > the edge-boundary tests would be enough? That is what the random tests are doing. One could massage them into using shorter strings (already a parameter), selecting fewer lengths from the middle (a simple code modification), and/or doing fewer edge-boundary tests (a simple code modification), of course. > That can't be more than a few hundred - which is few enough to run each > as a separate CppUnit case. I have not studied whether doing so will reduce the number of bugs found. I only studied that going in the opposite direction (more test cases) does not seem to help much. Please note that it is difficult to know for sure because while many test cases find bugs, it is difficult to isolate independent/different bugs without actually fixing SBuf. Why is "a few hundred" is OK as a separate CppUnit case but "a few hundred thousands" is not OK? I am _not_ trying to argue either way, just curious what the criteria for sizing CppUnit cases are, since I know so little about CppUnit... If more CppUnit cases are needed, I am sure it would be possible to parameterize the current code to generate more specific test cases. After repeating such "focused" generation inside each appropriate CppUnit case, one would end up with multiple CppUnit test cases covering approximately the same input space. > Also, using fuzzing with a random input leaves us open to the rare > occasions when a single byte octet value at a specific location is the > cause of a bug. Think \0 at octet 5 when doing 32-bit chunk comparisions > in some buggy implementation checking against !*p. That type of failure > appears intermittently and inspires confusion rather than a fix. Bugs notwithstanding, all random input is 100% reproducible so there should not be any intermittent behavior originating from the test cases themselves (at least as far as the same execution environment is concerned -- I do not know whether random() implementations yield different sequences on different platforms). HTH, Alex.
