On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:
On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
Hi all,
A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic, including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.
Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.
So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.
// jim
[1] https://review.openstack.org/#/c/220268/
What is the advantage of running another server like this over using
requests-mock (which is used by other OpenStack projects for testing
today)? The only difference here seems to be that you actually execute
requests code in one case and not in the other.
Requests-mock debugging when things go wrong seems a bit simpler.
This is less about twisted and more about trying to not introduce yet
another way to mock code in the tree that people need to understand.
-Sean
We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.
I don't really see the point of a separate project like Mimic that has a
whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
RAX-specific) API services. It's just a great way to introduce a larger
surface area for bugs to creep in -- since you have to keep the Mimic
interfaces up to date with the real interfaces. Better to keep something
like this -- if it is TRULY needed -- in-tree with the API service itself,
so that the chances of divergence are reduced. This is similar to the
fakevirt driver in Nova. It's in tree for good reason: when someone changes
the virt driver interface, the fakevirt driver goes boom and needs to be
changed in a corresponding fashion in the same patch.
I tend to think our REST APIs are much more stable than internal python
APIs, and so we can manage the divergence much easier.
Also, mimic can simulate:
* old versions (less needed with all the microversion stuff)
* old bugs that were shipped
* misconfigurations
* networking errors
* the passage of time (including timeouts)
We probably don't want to keep a catalog of old bugs and misconfigurations in
tree forever.
What value does a functional test against an HTTP API service that does
nothing (other than introduce greater surface area for bugs) actually offer
over unit tests anyway?
Testing the full stack of the client instead of mocking the bottom
half of requests is a big one.
Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
https://www.youtube.com/watch?v=HKUUQme3dwA
OK, I just watched that. Sorry, still don't see the value that Mimic
provides over unit testing the client interfaces and mocking out the
HTTP payloads so you have strict control over the expectations.
The problem that Glyph noted in the video about the unit test that
mocked out os.chdir to improperly return a value isn't solved whatsoever
by Mimic, so I find it odd that that example was used in discussing the
value of Mimic. Bad mocks are just that: bad mocks. The same false
positive failure could easily be introduced with a typo in the "metadata
injection" that Mimic does to inject failures into the system.
Mimic isn't testing anything on the server side at all so I'm not sure
why folks call it "integration testing". It isn't testing the
integration of anything at all. All it enables is multi-language client
library testing, and see my response to Ben, the surface area it
introduces for bugs in the test platform itself in my opinion outweigh
the multi-language value it might have.
Final question on this... if Mimic is *only* for testing clients, why
not make it just a dependency for python-ironicclient and not ironic itself?
Best,
-jay
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
