On 06/05/2014 18:58, Dicebot wrote:
On Tuesday, 6 May 2014 at 15:54:30 UTC, Bruno Medeiros wrote:
But before we continue the discussion, we are missing am more basic
assumption here: Do we want D to have a Unit-Testing facility, or a
Testing facility?? In other words, do we want to be able to write
automated tests that are Integration tests or just Unit tests? Because
if we go with this option of making D unittest blocks run in parallel,
we kill the option of them supporting Integration Tests. I don't think
this is good.
These days I often find myself leaning towards writing mostly
integration tests with only limited amount of unit tests. But writing
good integration test is very different from writing good unit test and
usually implies quite lot of boilerplate. Truth is D does not currently
have any higher-level facility at all. It has an _awesome_ unit test
facility which gets often misused for writing sloppy integration tests.
Indeed: I also find myself writing more integration tests than unit
tests, at least in the way I consider an integration tests to be (in
some cases the distinction between integration and unit test may not be
very easy or clear, IMO).
I'd love to keep existing facility as-is and think about providing good
library augmentation for any sort of higher level approach.
The unittest block is enough right now to support integration tests. To
support common test fixture setup (akin to @Before and @After in xUnit),
perhaps some syntax sugar could be added, although with D's language
facilities (meta-programming, functional constructs, scope statements,
etc.), we can do pretty well with existing functionality already.
Good integration test is very different. It has certain assumptions
about initial system state and notifies user if those are not met. It
can take ages to run and can test real-world situations. It is not
supposed to be run implicitly and frequently. You don't want to keep
your integration tests inline because of amount of boilerplate code
those usually need.
They are somewhat different. I wouldn't say very different. I don't
agree integration tests usually take ages to run. Some of them can run
fairly fast too, and are executed as frequently as unit tests.
In DDT for example I always run unit tests the same time as integration
tests. As I said, I don't find it useful to have a strict distinction
between those.
Rather, if I want to run a subset of tests, usually what I filter on is
running only the tests of a certain plugin (DDT has 3 plugins), Java
package, or Java class.
Additionally the parser tests can be run in what I call "Lite" mode,
which instead of running the full test suite, skips some of the
heavyweight, parameterized tests, to make the suite run faster. Most of
the cases are generated from templates, others are blind mass parse
tests, such as parsing all source modules in Phobos. But what "Lite"
mode cuts is not Integration tests, but rather the input sets of
parameterized tests.
As for keeping integration tests inline or not, yeah, you are likely to
prefer putting them in a separate file. Doesn't mean we need a different
language construct other than the unittest block for that.
I see no good in trying to unite those very different beasts and my
experience with existing test libraries has been very unpleasant in that
regard.
What test libraries/frameworks have you used?
--
Bruno Medeiros
https://twitter.com/brunodomedeiros
