On 08/01/16 04:49 AM, Julien Danjou wrote:
On Thu, Jan 07 2016, Jay Pipes wrote:
OK, I just watched that. Sorry, still don't see the value that Mimic provides
over unit testing the client interfaces and mocking out the HTTP payloads so
you have strict control over the expectations.
The problem that Glyph noted in the video about the unit test that mocked out
os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I
find it odd that that example was used in discussing the value of Mimic. Bad
mocks are just that: bad mocks. The same false positive failure could easily be
introduced with a typo in the "metadata injection" that Mimic does to inject
failures into the system.
Mimic isn't testing anything on the server side at all so I'm not sure why
folks call it "integration testing". It isn't testing the integration of
anything at all. All it enables is multi-language client library testing, and
see my response to Ben, the surface area it introduces for bugs in the test
platform itself in my opinion outweigh the multi-language value it might have.
I completely agree with what Jay points out here.
To illustrate with what we do in the Telemetry team: in order test
gnocchiclient, we pip install gnocchi¹, configure it briefly and use the
client against that freshly installed server. It works very well, and
we're sure we test against real data. That's what I would call
integration testing.
We could also easily test against older version of the Gnocchi server by
pip install-ing an older version if we wanted.
This has way more value than mocking the whole HTTP server and crossing
fingers we did a good job mimicking it. Oh, and it's also actually less
work. :)
¹ https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh
this happens with aodhclient as well. the one caveat i would mention is
that it doesn't really allow for test isolation which requires you to
put a bit more thought into your tests.
cheers,
--
gord
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
