On Thu, 16 Oct 2025 at 23:52, Martin D Kealey <[email protected]> wrote:
> > On Tue, 7 Oct 2025 at 23:55, Alexander Bantyev <[email protected]> > wrote: > >> Previously if a test produced a diff, the only way to fix it was to >> inspect the test output. >> > > And in some cases, it's non-trivial to figure out which test input > corresponds to which expected output. > (I'm working on a patch to rationalise the test harness framework, which > will make this a *lot* simpler.) > Here's my work on a test harness: https://github.com/kurahaupo/bash/tree/test-harness. Chet: I started this before your recent changes in tests/*, many of which I'd already included, but I've manually checked and updated it to include your additional changes. The primary intention is to allow tests to be run in parallel, which requires them to be properly isolated, and detached from the controlling tty. A secondary effect is that the new test framework is rather more picky than the old one: it checks stdout & stderr separately, and it checks exit statuses, unless there are additional controls to suppress these on a per-test basis. (For jobs where the relative order of stdout & stderr is meaningful, “exec 2>&1” suffices to combine them.) I've essentially rewritten tests/run-all, moving as much common logic as possible into tests/run-all. This approach makes the individual tests simpler, while run-all itself is more complicated. It now provides a framework that can validate stdout, stderr, and the exit status for each test. As a result most of tests/run-* can be thrown away, leaving just tests/*.tests. I've broken out most (but not all) tests/*.sub so that they're as separate test jobs; this has necessitated renaming many files, but their contents are unchanged and “git diff” reflects this. The test/*.right files are split accordingly and (as noted above) split into stdout and stderr. (This is actually where most of my effort is going.) When all old-style tests have been converted, support for them can be dropped, so that the overall change in code size will be minimal. More details of the new structure are in tests/README <https://github.com/kurahaupo/bash/blob/test-harness/tests/README>. -Martin PS: run-gprof is currently broken, so don't copy it just yet.
