yes Jay, i have -W on the bot proposed merge, pending this discussion: https://review.openstack.org/#/c/263592/
-- Dims On Fri, Jan 8, 2016 at 2:08 PM, Jay Pipes <[email protected]> wrote: > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: >> >> On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote: >>> >>> On 01/07/2016 06:42 PM, Jim Rollenhagen wrote: >>>> >>>> On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote: >>>>> >>>>> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: >>>>>> >>>>>> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: >>>>>>> >>>>>>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> A change to global-requirements[1] introduces mimic, which is an >>>>>>>> http >>>>>>>> server that can mock various APIs, including nova and ironic, >>>>>>>> including >>>>>>>> control of error codes and timeouts. The ironic team plans to use >>>>>>>> this >>>>>>>> for testing python-ironicclient without standing up a full ironic >>>>>>>> environment. >>>>>>>> >>>>>>>> Here's the catch - mimic is built on twisted. I know twisted was >>>>>>>> previously removed from OpenStack (or at least people said "pls no", >>>>>>>> I >>>>>>>> don't know the full history). We didn't intend to stealth-introduce >>>>>>>> twisted back into g-r, but it was pointed out to me that it may >>>>>>>> appear >>>>>>>> this way, so here I am letting everyone know. lifeless pointed out >>>>>>>> that >>>>>>>> when tests are failing, people may end up digging into mimic or >>>>>>>> twisted >>>>>>>> code, which most people in this community aren't familiar with >>>>>>>> AFAIK, >>>>>>>> which is a valid point though I hope it isn't required often. >>>>>>>> >>>>>>>> So, the primary question here is: do folks have a problem with >>>>>>>> adding >>>>>>>> twisted here? We're holding off on Ironic changes that depend on >>>>>>>> this >>>>>>>> until this discussion has happened, but aren't reverting the g-r >>>>>>>> change >>>>>>>> until we decide one way or another. >>>>>>>> >>>>>>>> // jim >>>>>>>> >>>>>>>> [1] https://review.openstack.org/#/c/220268/ >>>>>>> >>>>>>> >>>>>>> What is the advantage of running another server like this over using >>>>>>> requests-mock (which is used by other OpenStack projects for testing >>>>>>> today)? The only difference here seems to be that you actually >>>>>>> execute >>>>>>> requests code in one case and not in the other. >>>>>>> >>>>>>> Requests-mock debugging when things go wrong seems a bit simpler. >>>>>>> >>>>>>> This is less about twisted and more about trying to not introduce yet >>>>>>> another way to mock code in the tree that people need to understand. >>>>>>> >>>>>>> -Sean >>>>>> >>>>>> >>>>>> We'd be using this for functional tests, not unit, where we can't >>>>>> really >>>>>> inject mocks. The idea is that we could run a full functional suite >>>>>> against either mimic or a full ironic environment, just by changing a >>>>>> test setting. >>>>> >>>>> >>>>> I don't really see the point of a separate project like Mimic that has >>>>> a >>>>> whole bunch of reimplementations (mocked out) of all sorts of OpenStack >>>>> (and >>>>> RAX-specific) API services. It's just a great way to introduce a larger >>>>> surface area for bugs to creep in -- since you have to keep the Mimic >>>>> interfaces up to date with the real interfaces. Better to keep >>>>> something >>>>> like this -- if it is TRULY needed -- in-tree with the API service >>>>> itself, >>>>> so that the chances of divergence are reduced. This is similar to the >>>>> fakevirt driver in Nova. It's in tree for good reason: when someone >>>>> changes >>>>> the virt driver interface, the fakevirt driver goes boom and needs to >>>>> be >>>>> changed in a corresponding fashion in the same patch. >>>> >>>> >>>> I tend to think our REST APIs are much more stable than internal python >>>> APIs, and so we can manage the divergence much easier. >>>> >>>> Also, mimic can simulate: >>>> >>>> * old versions (less needed with all the microversion stuff) >>>> * old bugs that were shipped >>>> * misconfigurations >>>> * networking errors >>>> * the passage of time (including timeouts) >>>> >>>> We probably don't want to keep a catalog of old bugs and >>>> misconfigurations in >>>> tree forever. >>>> >>>>> What value does a functional test against an HTTP API service that does >>>>> nothing (other than introduce greater surface area for bugs) actually >>>>> offer >>>>> over unit tests anyway? >>>> >>>> >>>> Testing the full stack of the client instead of mocking the bottom >>>> half of requests is a big one. >>>> >>>> Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening. >>>> https://www.youtube.com/watch?v=HKUUQme3dwA >>> >>> >>> OK, I just watched that. Sorry, still don't see the value that Mimic >>> provides over unit testing the client interfaces and mocking out the HTTP >>> payloads so you have strict control over the expectations. >>> >>> The problem that Glyph noted in the video about the unit test that mocked >>> out os.chdir to improperly return a value isn't solved whatsoever by >>> Mimic, >>> so I find it odd that that example was used in discussing the value of >>> Mimic. Bad mocks are just that: bad mocks. The same false positive >>> failure >>> could easily be introduced with a typo in the "metadata injection" that >>> Mimic does to inject failures into the system. >>> >>> Mimic isn't testing anything on the server side at all so I'm not sure >>> why >>> folks call it "integration testing". It isn't testing the integration of >>> anything at all. All it enables is multi-language client library testing, >>> and see my response to Ben, the surface area it introduces for bugs in >>> the >>> test platform itself in my opinion outweigh the multi-language value it >>> might have. >> >> >> FWIW, I've only been calling it functional testing. I also don't care >> about the multi-language parts here. The sole purpose we have for mimic >> (so far) is to functionally test python-ironicclient without mocking the >> world in ironicclient. >> >>> Final question on this... if Mimic is *only* for testing clients, why not >>> make it just a dependency for python-ironicclient and not ironic itself? >> >> >> We haven't made it a dep for anything yet, only added to g-r. > > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that > makes functionally any difference, though (pun intended). > >> However, now that you mention that, a really ambitious goal would be to >> add a rabbit interface to mimic, and functionally test the API server >> (that it sends the right messages, etc). Another would be to mimic >> (Neutron, Glance) and test Ironic by itself. > > > So you would reimplement AMQP communication protocols using an in-memory > data store for queues. Sounds like an even greater surface area for bugs to > be introduced. > >> Last, I frankly don't understand why there's >> such heavy opposition to the ironic team using an additional tool for >> testing. > > > Since you asked, I'll be blunt. This isn't a personal attack on you, Jim, > though. > > a) Because it fractures the testing and QA processes used by upstream > contributors that work on OpenStack projects by requiring them to learn > another system -- and one that potentially would require them to understand > a whole new surface area for potential bugs > > b) Because it represents yet another RAX-driven divergence in the QA space. > CloudCafe took essentially all of the RAX folks that were at one point > working on Tempest and upstream QA and siloed them into a totally different > organization, in true RAX fashion. Instead of pulling the OpenStack QA > community along together, RAX QA continues to just do its own thing and > there's still bitterness on the tips of tongues. > > <snip> > >> There was *more* than sufficient time for "I don't think this is useful" >> to be posted on the review. > > > Sorry, this was the first I'd heard of it all. > >> If anyone has a hard technical reason why >> >> mimic should not be used to test an OpenStack project, I'm happy to >> listen. Otherwise I'd like to work on actually getting things done. > > > I've listed my hard technical reason: it introduces, IMHO, too large of a > surface area for bugs to creep in to the testing process versus the benefit > it provides. > > Best, > -jay > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
