David Storrs wrote on 11/8/18 11:10 AM:
Will test successes show output as well? For me that's the hard stop
on using RackUnit -- the fact that it shows nothing on success means
that there's no way to tell the difference between "everything ran and
passed", "some things ran and passed but some things that should have
run did not", and "nothing ran".
I've felt the same about this -- it still bothers me a little.
(Examples: In DrRacket, there's always a possibility that test-running
has been disabled in a preference setting that's buried in a
nonintuitive place, several hops deep in UI navigation, not a top-level
indicator, but that someone might turn off accidentally when trying to
adjust a different behavior. Or, on the command line, you might not
realize you're in the wrong directory to pick up the tests you're
expecting. Or maybe you'd like at least a crude idea of how much tests
are running for a bit of unfamiliar code.)
Perhaps the difference between "test failures" and "code errors" here
(neither of which you'll see in DrRacket, if there's not a problem) is
that, when you run code as a program, there is usually some other
behavior you can see, to know that at least some code has been
processed. Not so for passing tests.
My original portable Scheme test framework always printed a full log,
including ending with a summary like:
;;; END "Foo Station" TESTS: FAILED
;;; (Total: 4 Passed: 3 Failed: 1)
My later Racket-specific test framework, Overeasy, used the logger for
non-test-failure messages. But probably the Racket logs are not
normally visible to most programmers, so this isn't great.
Absent a richer test metadata mechanism in DrRacket and `raco test`, I'm
not sure how to do this better for both DrRacket and command line.
So... Would the following simple two-part approach (special-casing
DrRacket) work, or does someone have a better idea?
* When tests are under current DrRacket, we could use stdout for a
one-line successful tests summary message, and stderr for a one-line
tests failure summary message of the same format. The different streams
are so that we avoid misusing the HCI effect that DrRacket achieves when
it colors errors in angry red. (I think this doesn't break much, since
programs under DrRacket won't be using stdout and stderr for old Unix
"software tools" composition anyway.)
* When tests are run under `raco test`, we could use either stdout or
stderr for the one-line summary message. The question then is whether
any CI-like frameworks that run tests like `raco test` currently
treating any output as failure (I'm not certain, offhand). We could
always kludge an environment variable to inhibit the summary for unusual
situations, but I'd like to keep this kind of thing simple and clean.
An alternative to the above would be to expose test metadata from the
`test` submodules somehow, such as with "push" calls out akin to how
`error-display-handler` works, or by a convention of `test` submodules
providing a "pull" interface for programs like DrRacket to query.
Making that work requires changes to core Racket code. (And, as usual
with this kind of facility, probably half of everyone will think it's
being done wrong.) The simple stdout/stderr convention above,
special-casing DrRacket, OTOH, can be done easily, in a distributed and
non-disruptive way.
--
You received this message because you are subscribed to the Google Groups "Racket
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
