>
> ...
> Still, keeping the per-test stubbed return values in sync with the
> actual low-level behavior is the real challenge when using stubs.
> Perhaps a happy medium (without an ideal fake) would be to drive a
> simple fake with tables of known outputs. Then those tables could be
> vetted against the real thing. At my last job (manufacturing testing
> for SSDs) we actually did something like this for our code base. We
> would periodically run drives through the system and record the
> low-level inputs and outputs. Then for testing we would use a fake to
> play it back and make sure we got the same behavior
>
Just as a quick note, that was the goal of the
"environs/jujutest/livetests.go" suite. Specifically, it was intended to be
run against:
1) the testing server (ie fake) for a given provider
2) the live server for a given provider when supplied with a "-live" sort
of flag.
3) 1&2 for all providers that we have implemented.
The idea is that rather than just stubs that return canned input, you can
test that your fake server acts like the real remote production server, and
can be substituted for the production server in your testing.
And then ideally, that each Environ implementation (which should be called
Provider, fwiw) then conforms to the Environ interface{} abstraction, and
you can again substitute any of them in a given test.
Arguably at this point we could drop (1) in favor of a really good 2&3 and
running those against the live services in CI. As most of our testing uses
the 'dummy' provider anyway. Very few testts actually end up using the
goose testing server other than direct openstack provider tests. And while
I think those are still better against a Fake, I'd personally be willing to
figure out what the pragmatic call is.
One of the really nice bits is running non-live interface tests against
each provider implementation (done in pre-commit-to-trunk testing). Which
makes it less likely that we won't catch something until CI notices and
blocks commits to master.
John
=:->
--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev
