Thanks for the feedback. :) Your points are helpful and I just need
some clarification. Before that, though, I want to make sure we are
on the same page with the term "fake". Unfortunately testing
nomenclature is a bit muddled so you might be thinking of something
(even slightly) different. What gave me pause is you contrasted fakes
with doubles. So perhaps you could briefly explain what you mean by
both in the context of Go.
As for me, by "fake" I mean a struct that implements an interface with
essentially no logic other than to keep track of method calls and
facilitate controlling their return values explicitly. For examples
see the implementations for GCE and in `testing/fake.go`. Thus in
tests a fake may be used in place of the concrete implementation of
the interface that would be used in production.
The onus is on the test writer to populate the fake with the correct
return values such that they would match the expected behavior of the
concrete implementation. This contrasts with the approach (e.g. taken
by EC2) of plugging in an API server that simulates the necessary
portions of the low-level API. At that point you're testing your
simulated API server just as much as the code you actually want to
test. Furthermore, you still have to explicitly control the return
values. You're just doing it at a lower level.
Regardless, I'm convinced that testing needs to include both high
coverage via isolated unit tests and good enough coverage via "full
stack" integration tests. Essentially we have to ensure that layers
work together properly and that low-level APIs work the way we expect
(and don't change unexpectedly).
However, the discussion of when fakes are appropriate is somewhat
orthogonal to that of how to use fakes in Go (and juju) and what they
should look like there. I was hoping to focus on the latter, assuming
we agree that fakes are appropriate in at least some situations (which
it appears we do). :)
On Fri, Feb 13, 2015 at 5:22 AM, John Meinel <j...@arbash-meinel.com> wrote:
> High level thoughts:
>
> 1. Using an interface{} instead of patching function pointers sounds like a
> great thing
Exactly! :) Basically that was the original motivator.
> 2. Writing a provider via faking out rather than implementing a double
> means that it is really hard to run the cross-provider interface tests
> (environs/jujutest), which means we run into what we have with Azure, where
> *some* of the functions return a different error than the one we expect (see
> the recent 'juju restore' regression). Our interface tests aren't amazing,
> but if we can push more on that, then we can avoid having implementations
> leak through their abstractions.
Could you elaborate on this? I'm not terribly familiar with
environs/jujutest and hopefully we didn't need to take it into
consideration for the GCE provider. It sounds like you are saying we
should have tests that tie into the livetests code (I noticed at least
the EC2 tests that do so).
So I'm otherwise not clear on how our use of fakes in the GCE provider
has an impact. Wayne and I definitely want it be correct.
Furthermore, one goal we have is for the GCE provider to be the go-to
example for a provider implementation. If we're missing something
then we want to correct that. [1]
> 3. I do realize that time-to-something-working is a lot faster with a Fake
> implementation, but ongoing support and maintenance might pay off more. (I
> love how quickly the GCE provider was available, I'm concerned if we know
> that it actually acts like EC2/Openstack when it comes to edge cases.)
By "pay off more" I presume you mean "cost more", based on the
context. :) Are you saying that there is a uniform mechanism (via
environs/jujutest?) for checking all those edge cases for a provider?
If so, we certainly need to use it for the GCE provider. However, I'm
not clear on how our use of fakes relates to that, other than that it
implies we did not use the aforementioned uniform mechanism.
-eric
[1] We've discussed these motications on numerous occaions but Wayne
can chime in I've mischaracterized them. :)
--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev
