Julian Foad wrote on Thu, 16 Jan 2020 09:18 +00:00: > I inspected the debug prints whizzing by, and noted that it included > each of the different kinds of input cases I expected (e.g. empty > ranges, reversed ranges, duplicate ranges, overlapping ranges, etc.) and > that the results included examples of the failure modes I expected. > > Based on that observation, I altered the parameters such as the sizes of > the generated revision ranges and the repetition counts, until it > provided good coverage without excessive repetition. I aimed for "a few" > (more than one, less than hundreds) of occurrences of the same failure mode. > > That doesn't guarantee coverage of every possible edge case combination, > of course, but it gives me confidence that it's an effective test. I > admit it's an issue that that is not apparent to an observer. However, > anyone needing to debug again can enable the debug prints and see it.
Julian and I discussed this on IRC today; the conclusion is: - With the behaviour of the test with seed=0 having been manually reviewed, we're satisfied that the test achieves adequate coverage. - The properties that the magic numbers were chosen to satisfy will be documented in code comments. - Julian has further reviewed the code, leading to, e.g., r1872919. Thanks, Julian. Daniel
