> and they don't have to be unit tests. In some ways, that's the worst > option because they tend to be trivial functions being tested
Nobody's saying they should *all* be unit-tests. It's better to test components in isolation first, verify they do what they're supposed to, then follow up with more elaborate tests. That way, a regression is easier to spot and narrow down when a test failure occurs. BDD-style testing can help — I find tests easier to follow when they're written like plain English <https://travis-ci.org/file-icons/atom/jobs/627541381#L506>: Tree-view When a file entry is displayed When it matches an icon ✓ it shows the icon next to its name ✓ it removes the built-in icon class Archive-view When a zip-file is opened ✓ it shows icons for each of its contents When a file is displayed ✓ it shows an icon next to its filename ✓ it tests path-sensitive rules Once the simple stuff is covered, you can move on to running pixel-diffs against corpora of documents (which don't necessarily have to use a BDD-flavoured approach to testing). I'm something of a pedantic prick when it comes to airtight test coverage, so a lot of this mail probably stems from personal bias. On Mon, 6 Jan 2020 at 05:20, Ralph Corderoy <ra...@inputplus.co.uk> wrote: > Hi Ingo, > > > Werner wrote: > > > I think the proper way for testing groff would be to make it run > > > with a fuzzer, > > > > Yes, that would no doubt be useful. > ... > > is kind of orthogonal to developing a test suite, though. > ... > > A test suite, on the other hand, is most useful for making sure no > > regressions creep into the functionality, > > I think that's groff's pressing need: stopping regressions. Given what > I imagine is the part-time nature of contributions, to a large code > base, that implements a complex system of parts, e.g. the troff > language, I suspect a fix to more delicate bug may be quite high risk of > introducing a regression, and often this will be only apparent in the > PostScript, PDF, or pixels on the page being different; no warnings, > errors, or dumped core. > > That bug I referred to earlier is a good example. > > https://bugs.launchpad.net/ubuntu/+source/groff/+bug/42764 > > The footer printed with the -mm macros has changed position between > groff versions 1.17.2 and 1.18 > ... > .\" groff -mm -ma4 footermoves.tr >footermoves.ps > .\" > .PGFORM 21c-2i 29.7c-1i 1i 1 > .PH "'hl'hc'hr'" > .PF "'fl'fc'fr'" > .\" > .S +1 > .P > Footer not in a constant place across groff versions. > > If the `.S +1' is commented out the problem doesn't occur. > > Fixing known bugs, finding more bugs with fuzzing, implementing new > features, or adding something from another troff, are all less fun to do > if one's constantly concerned that there's not a good safety net to stop > faults getting released. And a buggy groff version or two means > documents have to start testing the troff implementation and its version > to implement workarounds. > > So I think regression tests come first, and they don't have to be unit > tests. In some ways, that's the worst option because they tend to be > trivial functions being tested; it's the interplay of the whole system > that matters. A corpus of documents and golden eyeballed output, say > ditroff, PostScript, or PNG to suit, would be a good start. Even if > it's just a script that compares renderings of all the documents shipped > with groff between two built-but-not-installed groffs. > > Do any of the other troffs have regression tests to our benefit? > > -- > Cheers, Ralph. > >