On Thu, Apr 28, 2016 at 11:44 PM Anastasia Macmood <
anastasia.macm...@canonical.com> wrote:
> I understand your desire for a quick turn around.
> But I question the value that you would get from running "fast" (short)
> tests - would this set include some fast running unit tests, integration
> tests and functional tests?
>
Yes.
> Simply because they have been identified as running quickly on some
> machines?
>
Yes. While absolute speed obviously varies based on hardware/load,
relative speed should be pretty constant across environments.
> How would you know if that "fast" run is comprehensive enough?
>
I know for sure that it would *not* be comprehensive. That's why we
wouldn't use it to gate landing code. It is only for developers as a quick
smoke test.
The alternative to this is running NO tests while I'm developing. Which is
what I do now (and probably what most of us do). I run tests in the one
package I'm currently editing, and usually even just a subset of those if
they're all full-stack tests. It's just not practical to do a full 15+
minute test run more often than once every few hours at most, unless
absolutely necessary.
> It sounds to me like you might as well say "let's run couple of tests
> randomly" and rely on these result until you commit...
>
If they're fast, sure. It's better than no tests. I run go vet
constantly. It doesn't find even 1% of the bugs in my code, but it finds
more than zero, and those ones I can fix before I even compile. go test
-short is not a test that will tell me if everything is definitely
correct. It's a test that'll tell me if something is definitely *incorrect*.
> I do not know what you will end up doing with your current dilemma. I
> second Andrew's suggestion as well \o/
> Developing short/long test distinctions and special processing for the
> tests that we maintain seems like a waste of our effort.
>
It's 3 very simple lines of code per slow test (or per suite if the suite
itself is slow). If that's too much, I could make it a single line of code
by writing a helper function. You'd just see this something like this:
func (s *MySuite) TestFullStackSomething(c *gc.C) {
testing.SkipIfShort(c)
// .... rest of test
}
The functionality for the -short flag and for skipping tests is already
built into the test infrastructure we use (both go test and gocheck).
There's no other effort required. It can be done incrementally as we're
doing other things. And if someone doesn't like it, doesn't trust it,
thinks it's a bad idea... it won't impact them at all. They can just
ignore this thread and everything will be exactly as it was (except for one
line of code they can ignore in some tests).
-Nate
--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev
