On Fri, Nov 2, 2018 at 3:38 PM Neil Van Dyke wrote:
> Anyone have info or preferences on what are good output conventions for
> Racket package embedded unit tests? (For use with all of: `raco test`,
> package catalog and other builds, DrRacket and other IDEs?)
>
> I intend to reimplement Overeasy
> ("https://www.neilvandyke.org/racket/overeasy/;) in a way that is even
> more lightweight in both ease-of-use and in implementation. The syntax
> and features will be very similar, for now (there's one innovative big
> feature I might add later), but what I'm not yet sure about is all the
> details of the output.
>
> For output, here's what I'm currently thinking:
>
> * Most test messages other than fails are sent to logger `debug@test`.
>
> * A test failing unexpectedly (or succeeding unexpectedly) results in a
> message to stderr.
>
Maybe instead use "info@test-pass" and "info@test-fail" ? These messages
are expected output, not debug output, so should probably not print at the
debug level. Also, putting them on separate loggers would offer
finer-grained control.
> * Should such a failing message also have some way of click navigating
> to the test failure in DrRacket, like for exceptions? (Note that the
> answer to the next might be involved.)
>
> * Should the first failing test raise `exn:fail?`? Or should
> `exn:fail?` be raised at the end of all a module's tests if any failed?
> Or should `exn:fail?` not be raised at all by the tests?
>
The way I've handled it in test-more is that:
- Failing tests emit a message, not an exception
- Tests that throw an exception will terminate the script unless wrapped
in a test-suite. In that case the test-suite converts the exception to a
test failure and the rest of the tests in the suite are not run but
subsequent tests / test suites will be.
> * Should a test *failing expectedly* (as marked with `:fail`) also
> result in a message to stderr? Maybe a `warning@test` logger is
> better? (Note that, IIRC, the package catalog will regard output as a
> failure of tests for the package. Which can remind us that there is
> `:fail`, but can also discourage the programmer/qa capturing and
> tracking the info about the expected-fail tests.)
>
How difficult would it be to make the package server determine success /
failure by the return value of the test script as opposed to the
presence/absence of output? In my opinion output is pretty orthogonal to
success or failure, and I can think of times when it's desirable to have a
package emit output on load. (e.g. when it's a deprecated version that is
about to be sunset)
> * For the test-failure messages to stderr, what textual format should
> they have? More like the current Racket exception format, or current
> RackUnit, or Overeasy? (Or is there a pressing need for a different
> format?)
>
I'm obviously a fan of test-more's conventions:
#lang racket
(require handy/test-more)
(ok #f "(ok #f) fails and returns #f")
(ok 7 "(ok 7) succeeds and returns 7")
(is 7 8 "(is 7 8) prints the failure message and returns 7")
(lives (thunk 'ok) "(lives (thunk 'ok)) succeeds and returns 'ok")
(test-suite
"test-suite example"
(ok 'yes "passes")
(ok #f "fails")
(diag "Note that test-suite returns void, so the rests of tests inside a
test-suite do not show up on the output. Consider this an incentive to use
test-suites")
(raise 'error)
)
output is:
NOT ok 1 - (ok #f) fails and returns #f
#f
ok 2 - (ok 7) succeeds and returns 7
7
NOT ok 3 - (is 7 8) prints the failure message and returns 7
Got: 7
Expected: 8
7
ok 4 - (lives (thunk 'ok)) succeeds and returns 'ok
'ok
(START test-suite: test-suite example)
ok 5 - passes
NOT ok 6 - fails
### (Note that test-suite returns void, so the rests of tests inside a
test-suite do not show
up on the output. Consider this an incentive to use test-suites.)
NOT ok 7 - Exception thrown! Test message: 'test-suite completed without
throwing uncaught
exception'. Exception: 'error'
Got: #f
Expected: #t
'error
Total tests passed so far: 3
Total tests failed so far: 4
(END test-suite: test-suite example)
WARNING: Neither (expect-n-tests N) nor (done-testing) was called. May not
have run all
tests.
Having the tests return their values is a deliberate choice, as it's useful
for doing conditionals and embedded testing. Having test-suite eat those
values so that they don't show up on the output is for aesthetics.
The warning at the end can be suppressed either by using (expect-n-tests
7) or (done-testing). [NB: 'test-suite' is a test, which is why this is 7
tests instead of 6]
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
For more options, visit
