Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Thu, Jan 14, 2016 at 11:00:16AM +1300, Robert Collins wrote: > On 8 January 2016 at 08:09, Jim Rollenhagenwrote: > > Hi all, > > > > A change to global-requirements[1] introduces mimic, which is an http > > server that can mock various APIs, including nova and ironic, including > > control of error codes and timeouts. The ironic team plans to use this > > for testing python-ironicclient without standing up a full ironic > > environment. > > Whee that was a bit of a thread. Sorry for kicking this off on IRC and > not chiming in until now. > > As I read it the following points were made: > - this is a new an different testing thing > - which has the same interface skew issues as regular mocks > - and devs may need to hack twisted to fix issues with it in future > - but it is being used to target local devs, not gate jobs [today] > > Interface skew can be handled a couple of ways. I'm a big fan of a > technique (that I *think* came out of Google) of having a test fake > layer which is substitutable by real things, so you can always run it > in 'slow mode' to make sure things still work with real components. > It's my understanding that with the way mimic is used, this can be > done: run the same tests with real e.g. devstack brought up > components. I think that that is good insurance, particularly if this > is being done periodically rather than > when-someone-remembers-or-has-an-issue. Indeed, this is mimic's intent, and the way it's been used elsewhere (e.g. by autoscale QA folks). > Another way to mitigate interface skew is to have the publisher of an > interface also publish a fast test fake for folk to use. I am also a > fan of doing that, but as we haven't done so yet, I think its ok to go > with what we have. If we couldn't run the same things with a real > implementation, I would be more worried - because mocks of moving > interfaces are fragile (e.g. mocking out oslo.config is fragile vs > mocking out os.unlink). So this made me think a bit and poke Glyph - turns out out-of-tree plugins are being worked on by mimic; might be worth investigating putting the mimic plugin for ironic in ironic's tree itself. > Personally, I'm fine hacking on Twisted code - I quite like it > (particularly good Twisted code :)) - and since folk seem ok with that > aspect, great - thank you for raising the discussion here. Totally; thanks for chiming in :) // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 8 January 2016 at 08:09, Jim Rollenhagenwrote: > Hi all, > > A change to global-requirements[1] introduces mimic, which is an http > server that can mock various APIs, including nova and ironic, including > control of error codes and timeouts. The ironic team plans to use this > for testing python-ironicclient without standing up a full ironic > environment. Whee that was a bit of a thread. Sorry for kicking this off on IRC and not chiming in until now. As I read it the following points were made: - this is a new an different testing thing - which has the same interface skew issues as regular mocks - and devs may need to hack twisted to fix issues with it in future - but it is being used to target local devs, not gate jobs [today] Interface skew can be handled a couple of ways. I'm a big fan of a technique (that I *think* came out of Google) of having a test fake layer which is substitutable by real things, so you can always run it in 'slow mode' to make sure things still work with real components. It's my understanding that with the way mimic is used, this can be done: run the same tests with real e.g. devstack brought up components. I think that that is good insurance, particularly if this is being done periodically rather than when-someone-remembers-or-has-an-issue. Another way to mitigate interface skew is to have the publisher of an interface also publish a fast test fake for folk to use. I am also a fan of doing that, but as we haven't done so yet, I think its ok to go with what we have. If we couldn't run the same things with a real implementation, I would be more worried - because mocks of moving interfaces are fragile (e.g. mocking out oslo.config is fragile vs mocking out os.unlink). Personally, I'm fine hacking on Twisted code - I quite like it (particularly good Twisted code :)) - and since folk seem ok with that aspect, great - thank you for raising the discussion here. I look forward to hearing how it goes for Ironic. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/08/2016 03:52 PM, Jim Rollenhagen wrote: > On Fri, Jan 08, 2016 at 03:39:51PM -0500, Jay Pipes wrote: >> >> No, nothing nefarious there. Sorry for letting my personal frustrations >> bubble over into this. >> >> I am not blocking anything from going forward and I definitely am not asking >> for a revert of any g-r patch. Nor am I trying to obstruct you in your >> governance of Ironic. >> >> I was just raising my concerns as an OpenStack citizen and getting my >> opinion out on paper. > > As you should; thanks for doing that. I'm eager to see how this goes. :) I think there is also a huge difference between how an effort like this goes in the first 6 months, and what it looks like in 2 or 3 years once none of the people originally working on it still are. Especially when you start talking about more complex API flows that require there to be state behind the scenes. The internal state associated with flows is where this tends to get into the weeds. An alternative approach would be making a fake / simulator driver like Jay suggested, and spawning all the ironic services in a single process using the oslo.messaging in memory driver and in mem sqlite db. This kind of approach is what we use in Nova functional tests to setup a whole stack fake cloud inside a single test process. At the end of the day it's the choice of the Ironic team. Just something to think really hard about. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/08/2016 12:56 PM, Jim Rollenhagen wrote: > On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote: >> Jim Rollenhagen wrote: >>> [...] >>> Here's the catch - mimic is built on twisted. I know twisted was >>> previously removed from OpenStack (or at least people said "pls no", I >>> don't know the full history). We didn't intend to stealth-introduce >>> twisted back into g-r, but it was pointed out to me that it may appear >>> this way, so here I am letting everyone know. lifeless pointed out that >>> when tests are failing, people may end up digging into mimic or twisted >>> code, which most people in this community aren't familiar with AFAIK, >>> which is a valid point though I hope it isn't required often. >> A bit of history with Twisted. >> >> Back in 2010 we decided we could not afford asking OpenStack developers to >> be familiar with multiple service architecture frameworks, and eventlet was >> chosen as the simplest framework to learn and debug. The best reference I >> found on this is still visible in the wiki: >> >> https://wiki.openstack.org/wiki/UnifiedServiceArchitecture >> >>> So, the primary question here is: do folks have a problem with adding >>> twisted here? We're holding off on Ironic changes that depend on this >>> until this discussion has happened, but aren't reverting the g-r change >>> until we decide one way or another. >> The only friction I see is how many developers would be expected to need to >> learn Twisted in order to complete their jobs. My understanding is that >> Twisted expertise could be needed to debug python-ironicclient functional >> tests, which makes the cost relatively limited. So if Mimic brings in a >> clear and significant benefit, I don't think its Twisted dependence should >> play that much against it. >> >> However, I agree with Sean and Jay that the benefit is unclear -- the few >> features that Mimic brings seem to be outweighed by the increased risk of >> introducing a delta between the implementation and the mock. If the main >> benefit is that it's used in other Rackspace projects for testing (like Ben >> said), I'm not sure that makes a compelling argument for the rest of the >> community... > No, that is not the main benefit, at all. Ben isn't involved in Ironic > and until now has had nothing to do with this work to add mimic here, so > I'm not sure where he got that impression from, or why he's speaking on > our behalf as to the goals here. I never claimed involvement in Ironic and from the outset noted my status on this list. I've only commented in so far as trying to show the value of mimic and similar tools. I'm sorry if there was any confusion in that with respect to ironic. That was certainly not my intent. > As pointed out before, the risk of a delta between the mocks in mimic > and reality is identical to the risk of a delta between the mocks in a > client's unit tests and reality, so I don't see a particular downside > there. Agreed. Ben __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 08/01/16 04:49 AM, Julien Danjou wrote: On Thu, Jan 07 2016, Jay Pipes wrote: OK, I just watched that. Sorry, still don't see the value that Mimic provides over unit testing the client interfaces and mocking out the HTTP payloads so you have strict control over the expectations. The problem that Glyph noted in the video about the unit test that mocked out os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I find it odd that that example was used in discussing the value of Mimic. Bad mocks are just that: bad mocks. The same false positive failure could easily be introduced with a typo in the "metadata injection" that Mimic does to inject failures into the system. Mimic isn't testing anything on the server side at all so I'm not sure why folks call it "integration testing". It isn't testing the integration of anything at all. All it enables is multi-language client library testing, and see my response to Ben, the surface area it introduces for bugs in the test platform itself in my opinion outweigh the multi-language value it might have. I completely agree with what Jay points out here. To illustrate with what we do in the Telemetry team: in order test gnocchiclient, we pip install gnocchi¹, configure it briefly and use the client against that freshly installed server. It works very well, and we're sure we test against real data. That's what I would call integration testing. We could also easily test against older version of the Gnocchi server by pip install-ing an older version if we wanted. This has way more value than mocking the whole HTTP server and crossing fingers we did a good job mimicking it. Oh, and it's also actually less work. :) ¹ https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh this happens with aodhclient as well. the one caveat i would mention is that it doesn't really allow for test isolation which requires you to put a bit more thought into your tests. cheers, -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote: > Jim Rollenhagen wrote: > >[...] > >Here's the catch - mimic is built on twisted. I know twisted was > >previously removed from OpenStack (or at least people said "pls no", I > >don't know the full history). We didn't intend to stealth-introduce > >twisted back into g-r, but it was pointed out to me that it may appear > >this way, so here I am letting everyone know. lifeless pointed out that > >when tests are failing, people may end up digging into mimic or twisted > >code, which most people in this community aren't familiar with AFAIK, > >which is a valid point though I hope it isn't required often. > > A bit of history with Twisted. > > Back in 2010 we decided we could not afford asking OpenStack developers to > be familiar with multiple service architecture frameworks, and eventlet was > chosen as the simplest framework to learn and debug. The best reference I > found on this is still visible in the wiki: > > https://wiki.openstack.org/wiki/UnifiedServiceArchitecture > > >So, the primary question here is: do folks have a problem with adding > >twisted here? We're holding off on Ironic changes that depend on this > >until this discussion has happened, but aren't reverting the g-r change > >until we decide one way or another. > > The only friction I see is how many developers would be expected to need to > learn Twisted in order to complete their jobs. My understanding is that > Twisted expertise could be needed to debug python-ironicclient functional > tests, which makes the cost relatively limited. So if Mimic brings in a > clear and significant benefit, I don't think its Twisted dependence should > play that much against it. > > However, I agree with Sean and Jay that the benefit is unclear -- the few > features that Mimic brings seem to be outweighed by the increased risk of > introducing a delta between the implementation and the mock. If the main > benefit is that it's used in other Rackspace projects for testing (like Ben > said), I'm not sure that makes a compelling argument for the rest of the > community... No, that is not the main benefit, at all. Ben isn't involved in Ironic and until now has had nothing to do with this work to add mimic here, so I'm not sure where he got that impression from, or why he's speaking on our behalf as to the goals here. As pointed out before, the risk of a delta between the mocks in mimic and reality is identical to the risk of a delta between the mocks in a client's unit tests and reality, so I don't see a particular downside there. Again, I think "benefit is unclear" isn't a valid reason to block this, so unless someone posts a revert we're going to move forward with this. // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Fri, Jan 08, 2016 at 03:39:51PM -0500, Jay Pipes wrote: > On 01/08/2016 02:52 PM, Jim Rollenhagen wrote: > >On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote: > >>On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: > >snippity snip snip > > > >>>We haven't made it a dep for anything yet, only added to g-r. > >> > >>According to Dims, not to g-r, but to u-c, right Dims? Not sure if that > >>makes functionally any difference, though (pun intended). > > > >Both. https://review.openstack.org/#/c/220268/ > > > >This thread was originally about twisted, which is added to u-c with the > >introduction of mimic. > > Got it, thanks. > > >>>However, now that you mention that, a really ambitious goal would be to > >>>add a rabbit interface to mimic, and functionally test the API server > >>>(that it sends the right messages, etc). Another would be to mimic > >>>(Neutron, Glance) and test Ironic by itself. > >> > >>So you would reimplement AMQP communication protocols using an in-memory > >>data store for queues. Sounds like an even greater surface area for bugs to > >>be introduced. > >> > >>>Last, I frankly don't understand why there's > >>>such heavy opposition to the ironic team using an additional tool for > >>>testing. > >> > >>Since you asked, I'll be blunt. This isn't a personal attack on you, Jim, > >>though. > >> > >>a) Because it fractures the testing and QA processes used by upstream > >>contributors that work on OpenStack projects by requiring them to learn > >>another system -- and one that potentially would require them to understand > >>a whole new surface area for potential bugs > > > >I don't think there's a large risk of needing to dig deep into mimic, > >and especially twisted. If this does prove to be a problem, I'm happy to > >remove it. However, we can't even explore what it would be like, or how > >hard we would depend on mimic, without mimic being in g-r. > > Sure, fair enough. > > >>b) Because it represents yet another RAX-driven divergence in the QA space. > >>CloudCafe took essentially all of the RAX folks that were at one point > >>working on Tempest and upstream QA and siloed them into a totally different > >>organization, in true RAX fashion. Instead of pulling the OpenStack QA > >>community along together, RAX QA continues to just do its own thing and > >>there's still bitterness on the tips of tongues. > > > >So, this isn't trying to replace anything. This is adding a different > >way to run functional tests, that is *much* faster than standing up a > >full ironic environment. This is helpful for developers that want to > >quickly run tests before posting them to gerrit, people that need to > >test in constrained environments, etc. > > I recognize it is much faster than standing up a real environment. And I > recognize that running faster client tests is a useful thing -- as long as > we can be confident that what is tested does not suffer from some of the > issues I identified earlier (syncing with real API and introduction of > greater surface area for bugs in the test platform itself). > > >I'm 100% against doing things like Rackspace did with tempest and > >cloudcafe, and I wouldn't be supporting this effort if I felt it was > >similar. Here's how this went: > > > >* Lekha started working on OnMetal QA, with a goal of doing some amount > > of upstream work as well. > > > >* She's previously worked on projects (like autoscale) that interact > > with OpenStack APIs, and seeing the need to test without a full nova > > environment, built mimic. > > > >* In talking with some of the other Ironic folks working on QA (from > > Intel, IBM, more), she presented mimic as something that may be useful > > for testing the client (and more). They (and I) agreed it was a neat > > idea worth trying. > > > >* Jay offered to help with the global-requirements patch as it's > > something he's done before, and did the review grinding here. > > > >* It finally landed, and lifeless asked me to bring up the Twisted > > conversation on the list. Note that this is not the "is mimic > > useful" conversation we're having now. Nobody remotely voiced > > concerns about using a new test tool until this thread happened. > > > >Please do let me know if any of that seems nefarious; I hope it doesn't. > > No, nothing nefarious there. Sorry for letting my personal frustrations > bubble over into this. > > I am not blocking anything from going forward and I definitely am not asking > for a revert of any g-r patch. Nor am I trying to obstruct you in your > governance of Ironic. > > I was just raising my concerns as an OpenStack citizen and getting my > opinion out on paper. As you should; thanks for doing that. I'm eager to see how this goes. :) // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
Excerpts from Jim Rollenhagen's message of 2016-01-07 11:09:32 -0800: > Hi all, > > A change to global-requirements[1] introduces mimic, which is an http > server that can mock various APIs, including nova and ironic, including > control of error codes and timeouts. The ironic team plans to use this > for testing python-ironicclient without standing up a full ironic > environment. > > Here's the catch - mimic is built on twisted. I know twisted was > previously removed from OpenStack (or at least people said "pls no", I > don't know the full history). We didn't intend to stealth-introduce > twisted back into g-r, but it was pointed out to me that it may appear > this way, so here I am letting everyone know. lifeless pointed out that > when tests are failing, people may end up digging into mimic or twisted > code, which most people in this community aren't familiar with AFAIK, > which is a valid point though I hope it isn't required often. > > So, the primary question here is: do folks have a problem with adding > twisted here? We're holding off on Ironic changes that depend on this > until this discussion has happened, but aren't reverting the g-r change > until we decide one way or another. > > // jim > > [1] https://review.openstack.org/#/c/220268/ > I looked through the Ironic specs for anything mentioning "mimic" (using the search box) and didn't find anything. Is there a plan or something describing what's happening that I can read as background for this? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote: > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: snippity snip snip > >We haven't made it a dep for anything yet, only added to g-r. > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that > makes functionally any difference, though (pun intended). Both. https://review.openstack.org/#/c/220268/ This thread was originally about twisted, which is added to u-c with the introduction of mimic. > >However, now that you mention that, a really ambitious goal would be to > >add a rabbit interface to mimic, and functionally test the API server > >(that it sends the right messages, etc). Another would be to mimic > >(Neutron, Glance) and test Ironic by itself. > > So you would reimplement AMQP communication protocols using an in-memory > data store for queues. Sounds like an even greater surface area for bugs to > be introduced. > > >Last, I frankly don't understand why there's > >such heavy opposition to the ironic team using an additional tool for > >testing. > > Since you asked, I'll be blunt. This isn't a personal attack on you, Jim, > though. > > a) Because it fractures the testing and QA processes used by upstream > contributors that work on OpenStack projects by requiring them to learn > another system -- and one that potentially would require them to understand > a whole new surface area for potential bugs I don't think there's a large risk of needing to dig deep into mimic, and especially twisted. If this does prove to be a problem, I'm happy to remove it. However, we can't even explore what it would be like, or how hard we would depend on mimic, without mimic being in g-r. > b) Because it represents yet another RAX-driven divergence in the QA space. > CloudCafe took essentially all of the RAX folks that were at one point > working on Tempest and upstream QA and siloed them into a totally different > organization, in true RAX fashion. Instead of pulling the OpenStack QA > community along together, RAX QA continues to just do its own thing and > there's still bitterness on the tips of tongues. So, this isn't trying to replace anything. This is adding a different way to run functional tests, that is *much* faster than standing up a full ironic environment. This is helpful for developers that want to quickly run tests before posting them to gerrit, people that need to test in constrained environments, etc. I'm 100% against doing things like Rackspace did with tempest and cloudcafe, and I wouldn't be supporting this effort if I felt it was similar. Here's how this went: * Lekha started working on OnMetal QA, with a goal of doing some amount of upstream work as well. * She's previously worked on projects (like autoscale) that interact with OpenStack APIs, and seeing the need to test without a full nova environment, built mimic. * In talking with some of the other Ironic folks working on QA (from Intel, IBM, more), she presented mimic as something that may be useful for testing the client (and more). They (and I) agreed it was a neat idea worth trying. * Jay offered to help with the global-requirements patch as it's something he's done before, and did the review grinding here. * It finally landed, and lifeless asked me to bring up the Twisted conversation on the list. Note that this is not the "is mimic useful" conversation we're having now. Nobody remotely voiced concerns about using a new test tool until this thread happened. Please do let me know if any of that seems nefarious; I hope it doesn't. > > > > >There was *more* than sufficient time for "I don't think this is useful" > >to be posted on the review. > > Sorry, this was the first I'd heard of it all. > > > If anyone has a hard technical reason why > >mimic should not be used to test an OpenStack project, I'm happy to > >listen. Otherwise I'd like to work on actually getting things done. > > I've listed my hard technical reason: it introduces, IMHO, too large of a > surface area for bugs to creep in to the testing process versus the benefit > it provides. Again, I don't think that's any larger of a surface area than mocks in client unit tests being incorrect, and we really won't know whether it causes problems in reality. // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800: > On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote: > > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: > snippity snip snip > > > >We haven't made it a dep for anything yet, only added to g-r. > > > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that > > makes functionally any difference, though (pun intended). > > Both. https://review.openstack.org/#/c/220268/ > > This thread was originally about twisted, which is added to u-c with the > introduction of mimic. > > > >However, now that you mention that, a really ambitious goal would be to > > >add a rabbit interface to mimic, and functionally test the API server > > >(that it sends the right messages, etc). Another would be to mimic > > >(Neutron, Glance) and test Ironic by itself. > > > > So you would reimplement AMQP communication protocols using an in-memory > > data store for queues. Sounds like an even greater surface area for bugs to > > be introduced. > > > > >Last, I frankly don't understand why there's > > >such heavy opposition to the ironic team using an additional tool for > > >testing. > > > > Since you asked, I'll be blunt. This isn't a personal attack on you, Jim, > > though. > > > > a) Because it fractures the testing and QA processes used by upstream > > contributors that work on OpenStack projects by requiring them to learn > > another system -- and one that potentially would require them to understand > > a whole new surface area for potential bugs > > I don't think there's a large risk of needing to dig deep into mimic, > and especially twisted. If this does prove to be a problem, I'm happy to > remove it. However, we can't even explore what it would be like, or how > hard we would depend on mimic, without mimic being in g-r. > > > b) Because it represents yet another RAX-driven divergence in the QA space. > > CloudCafe took essentially all of the RAX folks that were at one point > > working on Tempest and upstream QA and siloed them into a totally different > > organization, in true RAX fashion. Instead of pulling the OpenStack QA > > community along together, RAX QA continues to just do its own thing and > > there's still bitterness on the tips of tongues. > > So, this isn't trying to replace anything. This is adding a different > way to run functional tests, that is *much* faster than standing up a > full ironic environment. This is helpful for developers that want to > quickly run tests before posting them to gerrit, people that need to > test in constrained environments, etc. So it won't be used in the gate, just on local developer systems? Doug > > I'm 100% against doing things like Rackspace did with tempest and > cloudcafe, and I wouldn't be supporting this effort if I felt it was > similar. Here's how this went: > > * Lekha started working on OnMetal QA, with a goal of doing some amount > of upstream work as well. > > * She's previously worked on projects (like autoscale) that interact > with OpenStack APIs, and seeing the need to test without a full nova > environment, built mimic. > > * In talking with some of the other Ironic folks working on QA (from > Intel, IBM, more), she presented mimic as something that may be useful > for testing the client (and more). They (and I) agreed it was a neat > idea worth trying. > > * Jay offered to help with the global-requirements patch as it's > something he's done before, and did the review grinding here. > > * It finally landed, and lifeless asked me to bring up the Twisted > conversation on the list. Note that this is not the "is mimic > useful" conversation we're having now. Nobody remotely voiced > concerns about using a new test tool until this thread happened. > > Please do let me know if any of that seems nefarious; I hope it doesn't. > > > > > > > > > >There was *more* than sufficient time for "I don't think this is useful" > > >to be posted on the review. > > > > Sorry, this was the first I'd heard of it all. > > > > > If anyone has a hard technical reason why > > >mimic should not be used to test an OpenStack project, I'm happy to > > >listen. Otherwise I'd like to work on actually getting things done. > > > > I've listed my hard technical reason: it introduces, IMHO, too large of a > > surface area for bugs to creep in to the testing process versus the benefit > > it provides. > > Again, I don't think that's any larger of a surface area than mocks in client > unit tests being incorrect, and we really won't know whether it causes > problems in reality. > > // jim > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Fri, Jan 08, 2016 at 03:13:15PM -0500, Doug Hellmann wrote: > Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800: > > On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote: > > > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: > > snippity snip snip > > > > > >We haven't made it a dep for anything yet, only added to g-r. > > > > > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that > > > makes functionally any difference, though (pun intended). > > > > Both. https://review.openstack.org/#/c/220268/ > > > > This thread was originally about twisted, which is added to u-c with the > > introduction of mimic. > > > > > >However, now that you mention that, a really ambitious goal would be to > > > >add a rabbit interface to mimic, and functionally test the API server > > > >(that it sends the right messages, etc). Another would be to mimic > > > >(Neutron, Glance) and test Ironic by itself. > > > > > > So you would reimplement AMQP communication protocols using an in-memory > > > data store for queues. Sounds like an even greater surface area for bugs > > > to > > > be introduced. > > > > > > >Last, I frankly don't understand why there's > > > >such heavy opposition to the ironic team using an additional tool for > > > >testing. > > > > > > Since you asked, I'll be blunt. This isn't a personal attack on you, Jim, > > > though. > > > > > > a) Because it fractures the testing and QA processes used by upstream > > > contributors that work on OpenStack projects by requiring them to learn > > > another system -- and one that potentially would require them to > > > understand > > > a whole new surface area for potential bugs > > > > I don't think there's a large risk of needing to dig deep into mimic, > > and especially twisted. If this does prove to be a problem, I'm happy to > > remove it. However, we can't even explore what it would be like, or how > > hard we would depend on mimic, without mimic being in g-r. > > > > > b) Because it represents yet another RAX-driven divergence in the QA > > > space. > > > CloudCafe took essentially all of the RAX folks that were at one point > > > working on Tempest and upstream QA and siloed them into a totally > > > different > > > organization, in true RAX fashion. Instead of pulling the OpenStack QA > > > community along together, RAX QA continues to just do its own thing and > > > there's still bitterness on the tips of tongues. > > > > So, this isn't trying to replace anything. This is adding a different > > way to run functional tests, that is *much* faster than standing up a > > full ironic environment. This is helpful for developers that want to > > quickly run tests before posting them to gerrit, people that need to > > test in constrained environments, etc. > > So it won't be used in the gate, just on local developer systems? Initially, yes. I'd like to also add it to the gate at some point to make sure we don't break those interactions, though I could deal with a non-voting job just in case. // jim > > Doug > > > > > I'm 100% against doing things like Rackspace did with tempest and > > cloudcafe, and I wouldn't be supporting this effort if I felt it was > > similar. Here's how this went: > > > > * Lekha started working on OnMetal QA, with a goal of doing some amount > > of upstream work as well. > > > > * She's previously worked on projects (like autoscale) that interact > > with OpenStack APIs, and seeing the need to test without a full nova > > environment, built mimic. > > > > * In talking with some of the other Ironic folks working on QA (from > > Intel, IBM, more), she presented mimic as something that may be useful > > for testing the client (and more). They (and I) agreed it was a neat > > idea worth trying. > > > > * Jay offered to help with the global-requirements patch as it's > > something he's done before, and did the review grinding here. > > > > * It finally landed, and lifeless asked me to bring up the Twisted > > conversation on the list. Note that this is not the "is mimic > > useful" conversation we're having now. Nobody remotely voiced > > concerns about using a new test tool until this thread happened. > > > > Please do let me know if any of that seems nefarious; I hope it doesn't. > > > > > > > > > > > > > > >There was *more* than sufficient time for "I don't think this is useful" > > > >to be posted on the review. > > > > > > Sorry, this was the first I'd heard of it all. > > > > > > > If anyone has a hard technical reason why > > > >mimic should not be used to test an OpenStack project, I'm happy to > > > >listen. Otherwise I'd like to work on actually getting things done. > > > > > > I've listed my hard technical reason: it introduces, IMHO, too large of a > > > surface area for bugs to creep in to the testing process versus the > > > benefit > > > it provides. > > > > Again, I don't think that's any larger of a surface area than mocks in >
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
Excerpts from Jim Rollenhagen's message of 2016-01-08 12:30:33 -0800: > On Fri, Jan 08, 2016 at 03:13:15PM -0500, Doug Hellmann wrote: > > Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800: > > > On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote: > > > > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: > > > snippity snip snip > > > > > > > >We haven't made it a dep for anything yet, only added to g-r. > > > > > > > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that > > > > makes functionally any difference, though (pun intended). > > > > > > Both. https://review.openstack.org/#/c/220268/ > > > > > > This thread was originally about twisted, which is added to u-c with the > > > introduction of mimic. > > > > > > > >However, now that you mention that, a really ambitious goal would be to > > > > >add a rabbit interface to mimic, and functionally test the API server > > > > >(that it sends the right messages, etc). Another would be to mimic > > > > >(Neutron, Glance) and test Ironic by itself. > > > > > > > > So you would reimplement AMQP communication protocols using an in-memory > > > > data store for queues. Sounds like an even greater surface area for > > > > bugs to > > > > be introduced. > > > > > > > > >Last, I frankly don't understand why there's > > > > >such heavy opposition to the ironic team using an additional tool for > > > > >testing. > > > > > > > > Since you asked, I'll be blunt. This isn't a personal attack on you, > > > > Jim, > > > > though. > > > > > > > > a) Because it fractures the testing and QA processes used by upstream > > > > contributors that work on OpenStack projects by requiring them to learn > > > > another system -- and one that potentially would require them to > > > > understand > > > > a whole new surface area for potential bugs > > > > > > I don't think there's a large risk of needing to dig deep into mimic, > > > and especially twisted. If this does prove to be a problem, I'm happy to > > > remove it. However, we can't even explore what it would be like, or how > > > hard we would depend on mimic, without mimic being in g-r. > > > > > > > b) Because it represents yet another RAX-driven divergence in the QA > > > > space. > > > > CloudCafe took essentially all of the RAX folks that were at one point > > > > working on Tempest and upstream QA and siloed them into a totally > > > > different > > > > organization, in true RAX fashion. Instead of pulling the OpenStack QA > > > > community along together, RAX QA continues to just do its own thing and > > > > there's still bitterness on the tips of tongues. > > > > > > So, this isn't trying to replace anything. This is adding a different > > > way to run functional tests, that is *much* faster than standing up a > > > full ironic environment. This is helpful for developers that want to > > > quickly run tests before posting them to gerrit, people that need to > > > test in constrained environments, etc. > > > > So it won't be used in the gate, just on local developer systems? > > Initially, yes. I'd like to also add it to the gate at some point to > make sure we don't break those interactions, though I could deal with a > non-voting job just in case. OK, if that's the case then although I share Jay's concerns about bug surface area and keeping the APIs in sync, I think the risk is low enough that it would only annoy developers and not actually allow bugs to slip through testing. So, I'm OK with continuing with the experiment in Ironic. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/08/2016 02:52 PM, Jim Rollenhagen wrote: On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote: On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: snippity snip snip We haven't made it a dep for anything yet, only added to g-r. According to Dims, not to g-r, but to u-c, right Dims? Not sure if that makes functionally any difference, though (pun intended). Both. https://review.openstack.org/#/c/220268/ This thread was originally about twisted, which is added to u-c with the introduction of mimic. Got it, thanks. However, now that you mention that, a really ambitious goal would be to add a rabbit interface to mimic, and functionally test the API server (that it sends the right messages, etc). Another would be to mimic (Neutron, Glance) and test Ironic by itself. So you would reimplement AMQP communication protocols using an in-memory data store for queues. Sounds like an even greater surface area for bugs to be introduced. Last, I frankly don't understand why there's such heavy opposition to the ironic team using an additional tool for testing. Since you asked, I'll be blunt. This isn't a personal attack on you, Jim, though. a) Because it fractures the testing and QA processes used by upstream contributors that work on OpenStack projects by requiring them to learn another system -- and one that potentially would require them to understand a whole new surface area for potential bugs I don't think there's a large risk of needing to dig deep into mimic, and especially twisted. If this does prove to be a problem, I'm happy to remove it. However, we can't even explore what it would be like, or how hard we would depend on mimic, without mimic being in g-r. Sure, fair enough. b) Because it represents yet another RAX-driven divergence in the QA space. CloudCafe took essentially all of the RAX folks that were at one point working on Tempest and upstream QA and siloed them into a totally different organization, in true RAX fashion. Instead of pulling the OpenStack QA community along together, RAX QA continues to just do its own thing and there's still bitterness on the tips of tongues. So, this isn't trying to replace anything. This is adding a different way to run functional tests, that is *much* faster than standing up a full ironic environment. This is helpful for developers that want to quickly run tests before posting them to gerrit, people that need to test in constrained environments, etc. I recognize it is much faster than standing up a real environment. And I recognize that running faster client tests is a useful thing -- as long as we can be confident that what is tested does not suffer from some of the issues I identified earlier (syncing with real API and introduction of greater surface area for bugs in the test platform itself). I'm 100% against doing things like Rackspace did with tempest and cloudcafe, and I wouldn't be supporting this effort if I felt it was similar. Here's how this went: * Lekha started working on OnMetal QA, with a goal of doing some amount of upstream work as well. * She's previously worked on projects (like autoscale) that interact with OpenStack APIs, and seeing the need to test without a full nova environment, built mimic. * In talking with some of the other Ironic folks working on QA (from Intel, IBM, more), she presented mimic as something that may be useful for testing the client (and more). They (and I) agreed it was a neat idea worth trying. * Jay offered to help with the global-requirements patch as it's something he's done before, and did the review grinding here. * It finally landed, and lifeless asked me to bring up the Twisted conversation on the list. Note that this is not the "is mimic useful" conversation we're having now. Nobody remotely voiced concerns about using a new test tool until this thread happened. Please do let me know if any of that seems nefarious; I hope it doesn't. No, nothing nefarious there. Sorry for letting my personal frustrations bubble over into this. I am not blocking anything from going forward and I definitely am not asking for a revert of any g-r patch. Nor am I trying to obstruct you in your governance of Ironic. I was just raising my concerns as an OpenStack citizen and getting my opinion out on paper. Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
yes Jay, i have -W on the bot proposed merge, pending this discussion: https://review.openstack.org/#/c/263592/ -- Dims On Fri, Jan 8, 2016 at 2:08 PM, Jay Pipeswrote: > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: >> >> On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote: >>> >>> On 01/07/2016 06:42 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote: > > On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: >> >> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: >>> >>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: Hi all, A change to global-requirements[1] introduces mimic, which is an http server that can mock various APIs, including nova and ironic, including control of error codes and timeouts. The ironic team plans to use this for testing python-ironicclient without standing up a full ironic environment. Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. // jim [1] https://review.openstack.org/#/c/220268/ >>> >>> >>> What is the advantage of running another server like this over using >>> requests-mock (which is used by other OpenStack projects for testing >>> today)? The only difference here seems to be that you actually >>> execute >>> requests code in one case and not in the other. >>> >>> Requests-mock debugging when things go wrong seems a bit simpler. >>> >>> This is less about twisted and more about trying to not introduce yet >>> another way to mock code in the tree that people need to understand. >>> >>> -Sean >> >> >> We'd be using this for functional tests, not unit, where we can't >> really >> inject mocks. The idea is that we could run a full functional suite >> against either mimic or a full ironic environment, just by changing a >> test setting. > > > I don't really see the point of a separate project like Mimic that has > a > whole bunch of reimplementations (mocked out) of all sorts of OpenStack > (and > RAX-specific) API services. It's just a great way to introduce a larger > surface area for bugs to creep in -- since you have to keep the Mimic > interfaces up to date with the real interfaces. Better to keep > something > like this -- if it is TRULY needed -- in-tree with the API service > itself, > so that the chances of divergence are reduced. This is similar to the > fakevirt driver in Nova. It's in tree for good reason: when someone > changes > the virt driver interface, the fakevirt driver goes boom and needs to > be > changed in a corresponding fashion in the same patch. I tend to think our REST APIs are much more stable than internal python APIs, and so we can manage the divergence much easier. Also, mimic can simulate: * old versions (less needed with all the microversion stuff) * old bugs that were shipped * misconfigurations * networking errors * the passage of time (including timeouts) We probably don't want to keep a catalog of old bugs and misconfigurations in tree forever. > What value does a functional test against an HTTP API service that does > nothing (other than introduce greater surface area for bugs) actually > offer > over unit tests anyway? Testing the full stack of the client instead of mocking the bottom half of requests is a big one. Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening. https://www.youtube.com/watch?v=HKUUQme3dwA >>> >>> >>> OK, I just watched that. Sorry, still don't see the value that Mimic >>> provides over unit testing the client interfaces and mocking out the HTTP >>> payloads so you have strict control over the expectations. >>> >>> The problem that Glyph noted
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Fri, Jan 08, 2016 at 02:39:52PM -0500, Davanum Srinivas wrote: > yes Jay, i have -W on the bot proposed merge, pending this discussion: > https://review.openstack.org/#/c/263592/ Note that this doesn't actually block mimic from being in g-r, or being used. It's already in g-r. You'd need to revert https://review.openstack.org/#/c/220268/ . // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
Excerpts from Jim Rollenhagen's message of 2016-01-08 09:56:46 -0800: > On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote: > > Jim Rollenhagen wrote: > > >[...] > > >Here's the catch - mimic is built on twisted. I know twisted was > > >previously removed from OpenStack (or at least people said "pls no", I > > >don't know the full history). We didn't intend to stealth-introduce > > >twisted back into g-r, but it was pointed out to me that it may appear > > >this way, so here I am letting everyone know. lifeless pointed out that > > >when tests are failing, people may end up digging into mimic or twisted > > >code, which most people in this community aren't familiar with AFAIK, > > >which is a valid point though I hope it isn't required often. > > > > A bit of history with Twisted. > > > > Back in 2010 we decided we could not afford asking OpenStack developers to > > be familiar with multiple service architecture frameworks, and eventlet was > > chosen as the simplest framework to learn and debug. The best reference I > > found on this is still visible in the wiki: > > > > https://wiki.openstack.org/wiki/UnifiedServiceArchitecture > > > > >So, the primary question here is: do folks have a problem with adding > > >twisted here? We're holding off on Ironic changes that depend on this > > >until this discussion has happened, but aren't reverting the g-r change > > >until we decide one way or another. > > > > The only friction I see is how many developers would be expected to need to > > learn Twisted in order to complete their jobs. My understanding is that > > Twisted expertise could be needed to debug python-ironicclient functional > > tests, which makes the cost relatively limited. So if Mimic brings in a > > clear and significant benefit, I don't think its Twisted dependence should > > play that much against it. > > > > However, I agree with Sean and Jay that the benefit is unclear -- the few > > features that Mimic brings seem to be outweighed by the increased risk of > > introducing a delta between the implementation and the mock. If the main > > benefit is that it's used in other Rackspace projects for testing (like Ben > > said), I'm not sure that makes a compelling argument for the rest of the > > community... > > No, that is not the main benefit, at all. Ben isn't involved in Ironic > and until now has had nothing to do with this work to add mimic here, so > I'm not sure where he got that impression from, or why he's speaking on > our behalf as to the goals here. > > As pointed out before, the risk of a delta between the mocks in mimic > and reality is identical to the risk of a delta between the mocks in a > client's unit tests and reality, so I don't see a particular downside > there. > > Again, I think "benefit is unclear" isn't a valid reason to block this, > so unless someone posts a revert we're going to move forward with this. I have not made a decision myself about whether it's right to go ahead or not, but if we need to have a revert in place to continue the conversation, I'll do that. Please see https://review.openstack.org/#/c/265416/ as the patch to revert adding mimic. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/07/2016 07:38 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote: On 01/07/2016 06:42 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote: On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: Hi all, A change to global-requirements[1] introduces mimic, which is an http server that can mock various APIs, including nova and ironic, including control of error codes and timeouts. The ironic team plans to use this for testing python-ironicclient without standing up a full ironic environment. Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. // jim [1] https://review.openstack.org/#/c/220268/ What is the advantage of running another server like this over using requests-mock (which is used by other OpenStack projects for testing today)? The only difference here seems to be that you actually execute requests code in one case and not in the other. Requests-mock debugging when things go wrong seems a bit simpler. This is less about twisted and more about trying to not introduce yet another way to mock code in the tree that people need to understand. -Sean We'd be using this for functional tests, not unit, where we can't really inject mocks. The idea is that we could run a full functional suite against either mimic or a full ironic environment, just by changing a test setting. I don't really see the point of a separate project like Mimic that has a whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and RAX-specific) API services. It's just a great way to introduce a larger surface area for bugs to creep in -- since you have to keep the Mimic interfaces up to date with the real interfaces. Better to keep something like this -- if it is TRULY needed -- in-tree with the API service itself, so that the chances of divergence are reduced. This is similar to the fakevirt driver in Nova. It's in tree for good reason: when someone changes the virt driver interface, the fakevirt driver goes boom and needs to be changed in a corresponding fashion in the same patch. I tend to think our REST APIs are much more stable than internal python APIs, and so we can manage the divergence much easier. Also, mimic can simulate: * old versions (less needed with all the microversion stuff) * old bugs that were shipped * misconfigurations * networking errors * the passage of time (including timeouts) We probably don't want to keep a catalog of old bugs and misconfigurations in tree forever. What value does a functional test against an HTTP API service that does nothing (other than introduce greater surface area for bugs) actually offer over unit tests anyway? Testing the full stack of the client instead of mocking the bottom half of requests is a big one. Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening. https://www.youtube.com/watch?v=HKUUQme3dwA OK, I just watched that. Sorry, still don't see the value that Mimic provides over unit testing the client interfaces and mocking out the HTTP payloads so you have strict control over the expectations. The problem that Glyph noted in the video about the unit test that mocked out os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I find it odd that that example was used in discussing the value of Mimic. Bad mocks are just that: bad mocks. The same false positive failure could easily be introduced with a typo in the "metadata injection" that Mimic does to inject failures into the system. Mimic isn't testing anything on the server side at all so I'm not sure why folks call it "integration testing". It isn't testing the integration of anything at all. All it enables is multi-language client library testing, and see my response to Ben, the surface area it introduces for bugs in the test platform itself in my opinion outweigh the multi-language value it might have. FWIW, I've only been calling it functional testing. I also don't care about the multi-language parts here. The sole purpose we have for mimic (so far) is to functionally test python-ironicclient without mocking the world in
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Thu, Jan 07 2016, Jay Pipes wrote: > OK, I just watched that. Sorry, still don't see the value that Mimic provides > over unit testing the client interfaces and mocking out the HTTP payloads so > you have strict control over the expectations. > > The problem that Glyph noted in the video about the unit test that mocked out > os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I > find it odd that that example was used in discussing the value of Mimic. Bad > mocks are just that: bad mocks. The same false positive failure could easily > be > introduced with a typo in the "metadata injection" that Mimic does to inject > failures into the system. > > Mimic isn't testing anything on the server side at all so I'm not sure why > folks call it "integration testing". It isn't testing the integration of > anything at all. All it enables is multi-language client library testing, and > see my response to Ben, the surface area it introduces for bugs in the test > platform itself in my opinion outweigh the multi-language value it might have. I completely agree with what Jay points out here. To illustrate with what we do in the Telemetry team: in order test gnocchiclient, we pip install gnocchi¹, configure it briefly and use the client against that freshly installed server. It works very well, and we're sure we test against real data. That's what I would call integration testing. We could also easily test against older version of the Gnocchi server by pip install-ing an older version if we wanted. This has way more value than mocking the whole HTTP server and crossing fingers we did a good job mimicking it. Oh, and it's also actually less work. :) ¹ https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
Jim Rollenhagen wrote: [...] Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. A bit of history with Twisted. Back in 2010 we decided we could not afford asking OpenStack developers to be familiar with multiple service architecture frameworks, and eventlet was chosen as the simplest framework to learn and debug. The best reference I found on this is still visible in the wiki: https://wiki.openstack.org/wiki/UnifiedServiceArchitecture So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. The only friction I see is how many developers would be expected to need to learn Twisted in order to complete their jobs. My understanding is that Twisted expertise could be needed to debug python-ironicclient functional tests, which makes the cost relatively limited. So if Mimic brings in a clear and significant benefit, I don't think its Twisted dependence should play that much against it. However, I agree with Sean and Jay that the benefit is unclear -- the few features that Mimic brings seem to be outweighed by the increased risk of introducing a delta between the implementation and the mock. If the main benefit is that it's used in other Rackspace projects for testing (like Ben said), I'm not sure that makes a compelling argument for the rest of the community... -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/07/2016 06:28 PM, Jay Pipes wrote: > On 01/07/2016 06:12 PM, Ben Meyer wrote: >> On 01/07/2016 03:32 PM, Jay Pipes wrote: >>> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: > On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: >> Hi all, >> >> A change to global-requirements[1] introduces mimic, which is an >> http >> server that can mock various APIs, including nova and ironic, >> including >> control of error codes and timeouts. The ironic team plans to use >> this >> for testing python-ironicclient without standing up a full ironic >> environment. >> >> Here's the catch - mimic is built on twisted. I know twisted was >> previously removed from OpenStack (or at least people said "pls >> no", I >> don't know the full history). We didn't intend to stealth-introduce >> twisted back into g-r, but it was pointed out to me that it may >> appear >> this way, so here I am letting everyone know. lifeless pointed out >> that >> when tests are failing, people may end up digging into mimic or >> twisted >> code, which most people in this community aren't familiar with >> AFAIK, >> which is a valid point though I hope it isn't required often. >> >> So, the primary question here is: do folks have a problem with >> adding >> twisted here? We're holding off on Ironic changes that depend on >> this >> until this discussion has happened, but aren't reverting the g-r >> change >> until we decide one way or another. >> >> // jim >> >> [1] https://review.openstack.org/#/c/220268/ > > What is the advantage of running another server like this over using > requests-mock (which is used by other OpenStack projects for testing > today)? The only difference here seems to be that you actually > execute > requests code in one case and not in the other. > > Requests-mock debugging when things go wrong seems a bit simpler. > > This is less about twisted and more about trying to not introduce yet > another way to mock code in the tree that people need to understand. > > -Sean We'd be using this for functional tests, not unit, where we can't really inject mocks. The idea is that we could run a full functional suite against either mimic or a full ironic environment, just by changing a test setting. >>> >>> I don't really see the point of a separate project like Mimic that has >>> a whole bunch of reimplementations (mocked out) of all sorts of >>> OpenStack (and RAX-specific) API services. It's just a great way to >>> introduce a larger surface area for bugs to creep in -- since you have >>> to keep the Mimic interfaces up to date with the real interfaces. >>> Better to keep something like this -- if it is TRULY needed -- in-tree >>> with the API service itself, so that the chances of divergence are >>> reduced. This is similar to the fakevirt driver in Nova. It's in tree >>> for good reason: when someone changes the virt driver interface, the >>> fakevirt driver goes boom and needs to be changed in a corresponding >>> fashion in the same patch. >> A tool like my OpenStackInABox could certainly benefit from the models >> or services being provided by each project - aside from the complexity >> that that adds to the installation of installing all the dependencies >> related instead of just implementing a simple model with a file and >> sqlite backend that has minimal dependencies...but I digress as I >> haven't looked at how nova's fakevirt driver installs so may be that's >> not as big an issue. I'll certainly look at it for use in >> OpenStackInABox, but even there I'm aiming for a more complete scenario >> where you can interact with Keystone, Nova, Swift, etc...(e.g auth >> against a fake Keystone, use the token with the fake nova which >> validates it against the same fake Keystone instance, same with a fake >> Swift...). > > But your fake Keystone wouldn't be "authing" anything at all. Your > fake Nova wouldn't be "validating" anything at all. In my OpenStackInABox functionality, it does do authing. It generates and maintains in the model auth tokens, and can be configured to reject all tokens, etc. depending on your test. Same for my swift mock - it stores the data and generates responses just like real Swift - in accordance with the API documentation in both cases. > You aren't *functionally* testing anything of importance above if the > things you are testing aren't doing what they are supposed to do. But you are. Sure, you're not going to have the same timing, but it at least proves the code paths really well. It also reduces the necessary code to run the tests. > The only things you are functionally testing are the *clients* to > those fake HTTP services. And what you are *actually* validating the > client code for isn't *actually*
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/07/2016 06:12 PM, Ben Meyer wrote: On 01/07/2016 03:32 PM, Jay Pipes wrote: On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: Hi all, A change to global-requirements[1] introduces mimic, which is an http server that can mock various APIs, including nova and ironic, including control of error codes and timeouts. The ironic team plans to use this for testing python-ironicclient without standing up a full ironic environment. Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. // jim [1] https://review.openstack.org/#/c/220268/ What is the advantage of running another server like this over using requests-mock (which is used by other OpenStack projects for testing today)? The only difference here seems to be that you actually execute requests code in one case and not in the other. Requests-mock debugging when things go wrong seems a bit simpler. This is less about twisted and more about trying to not introduce yet another way to mock code in the tree that people need to understand. -Sean We'd be using this for functional tests, not unit, where we can't really inject mocks. The idea is that we could run a full functional suite against either mimic or a full ironic environment, just by changing a test setting. I don't really see the point of a separate project like Mimic that has a whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and RAX-specific) API services. It's just a great way to introduce a larger surface area for bugs to creep in -- since you have to keep the Mimic interfaces up to date with the real interfaces. Better to keep something like this -- if it is TRULY needed -- in-tree with the API service itself, so that the chances of divergence are reduced. This is similar to the fakevirt driver in Nova. It's in tree for good reason: when someone changes the virt driver interface, the fakevirt driver goes boom and needs to be changed in a corresponding fashion in the same patch. A tool like my OpenStackInABox could certainly benefit from the models or services being provided by each project - aside from the complexity that that adds to the installation of installing all the dependencies related instead of just implementing a simple model with a file and sqlite backend that has minimal dependencies...but I digress as I haven't looked at how nova's fakevirt driver installs so may be that's not as big an issue. I'll certainly look at it for use in OpenStackInABox, but even there I'm aiming for a more complete scenario where you can interact with Keystone, Nova, Swift, etc...(e.g auth against a fake Keystone, use the token with the fake nova which validates it against the same fake Keystone instance, same with a fake Swift...). But your fake Keystone wouldn't be "authing" anything at all. Your fake Nova wouldn't be "validating" anything at all. You aren't *functionally* testing anything of importance above if the things you are testing aren't doing what they are supposed to do. The only things you are functionally testing are the *clients* to those fake HTTP services. And what you are *actually* validating the client code for isn't *actually* the real HTTP API service -- it's a fake which can have its own surface area for bugginess -- which takes me back to my original question: what value does one get from *functional* tests of a client that calls a faked-out HTTP API versus *unit* tests of said client that simply uses requests-mock (or similar) to set the returned value of the HTTP API service to some expected value? What value does a functional test against an HTTP API service that does nothing (other than introduce greater surface area for bugs) actually offer over unit tests anyway? So to use Nova's fakevirt your project is limited to the same language as nova - python, correct? Nova's fakevirt driver simply does NOOP calls for most things and some light in-memory bookkeeping of resources that a normal virt driver would, in reality, consume from the host. It's a way we have to "test" launching many instances in Nova without actually having those instances consume resource on the host.
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Thu, Jan 7, 2016 at 3:01 PM, Jim Rollenhagenwrote: > We'd be using this for functional tests, not unit, where we can't really > inject mocks. The idea is that we could run a full functional suite > against either mimic or a full ironic environment, just by changing a > test setting. > I'm assuming that this would be for client functional tests because it wouldn't make sense for testing a server. How is the interface created? It seems like it would be possible for the mocked API to not match the actual API. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek www: http://dstanek.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/07/2016 03:32 PM, Jay Pipes wrote: > On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: >> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: >>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: Hi all, A change to global-requirements[1] introduces mimic, which is an http server that can mock various APIs, including nova and ironic, including control of error codes and timeouts. The ironic team plans to use this for testing python-ironicclient without standing up a full ironic environment. Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. // jim [1] https://review.openstack.org/#/c/220268/ >>> >>> What is the advantage of running another server like this over using >>> requests-mock (which is used by other OpenStack projects for testing >>> today)? The only difference here seems to be that you actually execute >>> requests code in one case and not in the other. >>> >>> Requests-mock debugging when things go wrong seems a bit simpler. >>> >>> This is less about twisted and more about trying to not introduce yet >>> another way to mock code in the tree that people need to understand. >>> >>> -Sean >> >> We'd be using this for functional tests, not unit, where we can't really >> inject mocks. The idea is that we could run a full functional suite >> against either mimic or a full ironic environment, just by changing a >> test setting. > > I don't really see the point of a separate project like Mimic that has > a whole bunch of reimplementations (mocked out) of all sorts of > OpenStack (and RAX-specific) API services. It's just a great way to > introduce a larger surface area for bugs to creep in -- since you have > to keep the Mimic interfaces up to date with the real interfaces. > Better to keep something like this -- if it is TRULY needed -- in-tree > with the API service itself, so that the chances of divergence are > reduced. This is similar to the fakevirt driver in Nova. It's in tree > for good reason: when someone changes the virt driver interface, the > fakevirt driver goes boom and needs to be changed in a corresponding > fashion in the same patch. A tool like my OpenStackInABox could certainly benefit from the models or services being provided by each project - aside from the complexity that that adds to the installation of installing all the dependencies related instead of just implementing a simple model with a file and sqlite backend that has minimal dependencies...but I digress as I haven't looked at how nova's fakevirt driver installs so may be that's not as big an issue. I'll certainly look at it for use in OpenStackInABox, but even there I'm aiming for a more complete scenario where you can interact with Keystone, Nova, Swift, etc...(e.g auth against a fake Keystone, use the token with the fake nova which validates it against the same fake Keystone instance, same with a fake Swift...). > What value does a functional test against an HTTP API service that > does nothing (other than introduce greater surface area for bugs) > actually offer over unit tests anyway? So to use Nova's fakevirt your project is limited to the same language as nova - python, correct? The main advantage of mimic is that it is language agnostic and multiple moving parts (f.e taskflow) can interact with it. $0.02 Ben __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote: > On 01/07/2016 06:42 PM, Jim Rollenhagen wrote: > >On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote: > >>On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: > >>>On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: > On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: > >Hi all, > > > >A change to global-requirements[1] introduces mimic, which is an http > >server that can mock various APIs, including nova and ironic, including > >control of error codes and timeouts. The ironic team plans to use this > >for testing python-ironicclient without standing up a full ironic > >environment. > > > >Here's the catch - mimic is built on twisted. I know twisted was > >previously removed from OpenStack (or at least people said "pls no", I > >don't know the full history). We didn't intend to stealth-introduce > >twisted back into g-r, but it was pointed out to me that it may appear > >this way, so here I am letting everyone know. lifeless pointed out that > >when tests are failing, people may end up digging into mimic or twisted > >code, which most people in this community aren't familiar with AFAIK, > >which is a valid point though I hope it isn't required often. > > > >So, the primary question here is: do folks have a problem with adding > >twisted here? We're holding off on Ironic changes that depend on this > >until this discussion has happened, but aren't reverting the g-r change > >until we decide one way or another. > > > >// jim > > > >[1] https://review.openstack.org/#/c/220268/ > > What is the advantage of running another server like this over using > requests-mock (which is used by other OpenStack projects for testing > today)? The only difference here seems to be that you actually execute > requests code in one case and not in the other. > > Requests-mock debugging when things go wrong seems a bit simpler. > > This is less about twisted and more about trying to not introduce yet > another way to mock code in the tree that people need to understand. > > -Sean > >>> > >>>We'd be using this for functional tests, not unit, where we can't really > >>>inject mocks. The idea is that we could run a full functional suite > >>>against either mimic or a full ironic environment, just by changing a > >>>test setting. > >> > >>I don't really see the point of a separate project like Mimic that has a > >>whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and > >>RAX-specific) API services. It's just a great way to introduce a larger > >>surface area for bugs to creep in -- since you have to keep the Mimic > >>interfaces up to date with the real interfaces. Better to keep something > >>like this -- if it is TRULY needed -- in-tree with the API service itself, > >>so that the chances of divergence are reduced. This is similar to the > >>fakevirt driver in Nova. It's in tree for good reason: when someone changes > >>the virt driver interface, the fakevirt driver goes boom and needs to be > >>changed in a corresponding fashion in the same patch. > > > >I tend to think our REST APIs are much more stable than internal python > >APIs, and so we can manage the divergence much easier. > > > >Also, mimic can simulate: > > > >* old versions (less needed with all the microversion stuff) > >* old bugs that were shipped > >* misconfigurations > >* networking errors > >* the passage of time (including timeouts) > > > >We probably don't want to keep a catalog of old bugs and misconfigurations in > >tree forever. > > > >>What value does a functional test against an HTTP API service that does > >>nothing (other than introduce greater surface area for bugs) actually offer > >>over unit tests anyway? > > > >Testing the full stack of the client instead of mocking the bottom > >half of requests is a big one. > > > >Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening. > >https://www.youtube.com/watch?v=HKUUQme3dwA > > OK, I just watched that. Sorry, still don't see the value that Mimic > provides over unit testing the client interfaces and mocking out the HTTP > payloads so you have strict control over the expectations. > > The problem that Glyph noted in the video about the unit test that mocked > out os.chdir to improperly return a value isn't solved whatsoever by Mimic, > so I find it odd that that example was used in discussing the value of > Mimic. Bad mocks are just that: bad mocks. The same false positive failure > could easily be introduced with a typo in the "metadata injection" that > Mimic does to inject failures into the system. > > Mimic isn't testing anything on the server side at all so I'm not sure why > folks call it "integration testing". It isn't testing the integration of > anything at all. All it enables is multi-language client library testing, > and see my
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote: > On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: > >On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: > >>On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: > >>>Hi all, > >>> > >>>A change to global-requirements[1] introduces mimic, which is an http > >>>server that can mock various APIs, including nova and ironic, including > >>>control of error codes and timeouts. The ironic team plans to use this > >>>for testing python-ironicclient without standing up a full ironic > >>>environment. > >>> > >>>Here's the catch - mimic is built on twisted. I know twisted was > >>>previously removed from OpenStack (or at least people said "pls no", I > >>>don't know the full history). We didn't intend to stealth-introduce > >>>twisted back into g-r, but it was pointed out to me that it may appear > >>>this way, so here I am letting everyone know. lifeless pointed out that > >>>when tests are failing, people may end up digging into mimic or twisted > >>>code, which most people in this community aren't familiar with AFAIK, > >>>which is a valid point though I hope it isn't required often. > >>> > >>>So, the primary question here is: do folks have a problem with adding > >>>twisted here? We're holding off on Ironic changes that depend on this > >>>until this discussion has happened, but aren't reverting the g-r change > >>>until we decide one way or another. > >>> > >>>// jim > >>> > >>>[1] https://review.openstack.org/#/c/220268/ > >> > >>What is the advantage of running another server like this over using > >>requests-mock (which is used by other OpenStack projects for testing > >>today)? The only difference here seems to be that you actually execute > >>requests code in one case and not in the other. > >> > >>Requests-mock debugging when things go wrong seems a bit simpler. > >> > >>This is less about twisted and more about trying to not introduce yet > >>another way to mock code in the tree that people need to understand. > >> > >>-Sean > > > >We'd be using this for functional tests, not unit, where we can't really > >inject mocks. The idea is that we could run a full functional suite > >against either mimic or a full ironic environment, just by changing a > >test setting. > > I don't really see the point of a separate project like Mimic that has a > whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and > RAX-specific) API services. It's just a great way to introduce a larger > surface area for bugs to creep in -- since you have to keep the Mimic > interfaces up to date with the real interfaces. Better to keep something > like this -- if it is TRULY needed -- in-tree with the API service itself, > so that the chances of divergence are reduced. This is similar to the > fakevirt driver in Nova. It's in tree for good reason: when someone changes > the virt driver interface, the fakevirt driver goes boom and needs to be > changed in a corresponding fashion in the same patch. I tend to think our REST APIs are much more stable than internal python APIs, and so we can manage the divergence much easier. Also, mimic can simulate: * old versions (less needed with all the microversion stuff) * old bugs that were shipped * misconfigurations * networking errors * the passage of time (including timeouts) We probably don't want to keep a catalog of old bugs and misconfigurations in tree forever. > What value does a functional test against an HTTP API service that does > nothing (other than introduce greater surface area for bugs) actually offer > over unit tests anyway? Testing the full stack of the client instead of mocking the bottom half of requests is a big one. Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening. https://www.youtube.com/watch?v=HKUUQme3dwA // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/07/2016 06:42 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote: On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: Hi all, A change to global-requirements[1] introduces mimic, which is an http server that can mock various APIs, including nova and ironic, including control of error codes and timeouts. The ironic team plans to use this for testing python-ironicclient without standing up a full ironic environment. Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. // jim [1] https://review.openstack.org/#/c/220268/ What is the advantage of running another server like this over using requests-mock (which is used by other OpenStack projects for testing today)? The only difference here seems to be that you actually execute requests code in one case and not in the other. Requests-mock debugging when things go wrong seems a bit simpler. This is less about twisted and more about trying to not introduce yet another way to mock code in the tree that people need to understand. -Sean We'd be using this for functional tests, not unit, where we can't really inject mocks. The idea is that we could run a full functional suite against either mimic or a full ironic environment, just by changing a test setting. I don't really see the point of a separate project like Mimic that has a whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and RAX-specific) API services. It's just a great way to introduce a larger surface area for bugs to creep in -- since you have to keep the Mimic interfaces up to date with the real interfaces. Better to keep something like this -- if it is TRULY needed -- in-tree with the API service itself, so that the chances of divergence are reduced. This is similar to the fakevirt driver in Nova. It's in tree for good reason: when someone changes the virt driver interface, the fakevirt driver goes boom and needs to be changed in a corresponding fashion in the same patch. I tend to think our REST APIs are much more stable than internal python APIs, and so we can manage the divergence much easier. Also, mimic can simulate: * old versions (less needed with all the microversion stuff) * old bugs that were shipped * misconfigurations * networking errors * the passage of time (including timeouts) We probably don't want to keep a catalog of old bugs and misconfigurations in tree forever. What value does a functional test against an HTTP API service that does nothing (other than introduce greater surface area for bugs) actually offer over unit tests anyway? Testing the full stack of the client instead of mocking the bottom half of requests is a big one. Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening. https://www.youtube.com/watch?v=HKUUQme3dwA OK, I just watched that. Sorry, still don't see the value that Mimic provides over unit testing the client interfaces and mocking out the HTTP payloads so you have strict control over the expectations. The problem that Glyph noted in the video about the unit test that mocked out os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I find it odd that that example was used in discussing the value of Mimic. Bad mocks are just that: bad mocks. The same false positive failure could easily be introduced with a typo in the "metadata injection" that Mimic does to inject failures into the system. Mimic isn't testing anything on the server side at all so I'm not sure why folks call it "integration testing". It isn't testing the integration of anything at all. All it enables is multi-language client library testing, and see my response to Ben, the surface area it introduces for bugs in the test platform itself in my opinion outweigh the multi-language value it might have. Final question on this... if Mimic is *only* for testing clients, why not make it just a dependency for python-ironicclient and not ironic itself? Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
-Original Message- From: David Stanek <dsta...@dstanek.com> Reply: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Date: January 7, 2016 at 18:14:23 To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [all] re-introducing twisted to global-requirements > On Thu, Jan 7, 2016 at 3:01 PM, Jim Rollenhagen > wrote: > > > We'd be using this for functional tests, not unit, where we can't really > > inject mocks. The idea is that we could run a full functional suite > > against either mimic or a full ironic environment, just by changing a > > test setting. > > > > I'm assuming that this would be for client functional tests because it > wouldn't make sense for testing a server. How is the interface created? It > seems like it would be possible for the mocked API to not match the actual > API. I agree. It's very easy for servers intending to mimic APIs and their behaviour to fall out of synchronization with the server they're attempting to mimic. -- Ian Cordasco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/07/2016 02:41 PM, Sean Dague wrote: > On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: >> Hi all, >> >> A change to global-requirements[1] introduces mimic, which is an http >> server that can mock various APIs, including nova and ironic, including >> control of error codes and timeouts. The ironic team plans to use this >> for testing python-ironicclient without standing up a full ironic >> environment. >> >> Here's the catch - mimic is built on twisted. I know twisted was >> previously removed from OpenStack (or at least people said "pls no", I >> don't know the full history). We didn't intend to stealth-introduce >> twisted back into g-r, but it was pointed out to me that it may appear >> this way, so here I am letting everyone know. lifeless pointed out that >> when tests are failing, people may end up digging into mimic or twisted >> code, which most people in this community aren't familiar with AFAIK, >> which is a valid point though I hope it isn't required often. >> >> So, the primary question here is: do folks have a problem with adding >> twisted here? We're holding off on Ironic changes that depend on this >> until this discussion has happened, but aren't reverting the g-r change >> until we decide one way or another. >> >> // jim >> >> [1] https://review.openstack.org/#/c/220268/ > What is the advantage of running another server like this over using > requests-mock (which is used by other OpenStack projects for testing > today)? The only difference here seems to be that you actually execute > requests code in one case and not in the other. > > Requests-mock debugging when things go wrong seems a bit simpler. > > This is less about twisted and more about trying to not introduce yet > another way to mock code in the tree that people need to understand. As an outsider and lurker...this is a fair point. I also maintain another project - StackInABox (and OpenStackInABox) - that resolves the issue in a slightly different way that utilizes requests-mock to do something similar, but closer to true unit tests. The main advantage of mimic (https://pypi.python.org/pypi/mimic) is that it can be independent of language framework (e.g you can use it for C++ or Java or Python2, Python3, or...) as its an independent process from your testing, and the tasks can be handled very asynchronously so it can mock out the responses for various tasks in a timely manner, and several parts of a system can interact together during a test. You're also utilizing a common mock of the entire system so multiple projects can benefit from the same code base. It's kind of integration-test but not quite - a good fit for projects where unit tests need multiple other components working together, e.g services that use taskflow would certainly benefit from using mimic. The disadvantage of mimic from what I can tell is how the tests are setup. But I'll leave that more to the mimic users and maintainers (OpenStack Poppy, Rackspace Autoscale, etc) to explain. For comparison: My StackInABox project provides a framework upon which OpenStackInABox builds to provide a common model so multiple projects benefit. I've used StackInABox on several of my own (non-OpenStack projects) and my unit tests (run by tox) have been able to be nearly complete integration tests. Aside from the slight use-case difference (as exemplified by Rackspace AutoScale), the main advantage of mimic over my OpenStackInABox is the momentum behind it due to its use by Rackspace AutoScale and OpenStack Poppy; I just don't have the resources to build out the models in OpenStackInABox efficiently or quickly (though I'd certainly welcome the help to do so). Advantage of the StackInABox approach is every unit test is self-contained and you don't have to change code to run the test in most cases (where you do, it's minimal) - just like with using requests-mock; but you can't have two services independently hit it - like with Rackspace AutoScale. HTH, Ben P.S here's my two projects: https://github.com/BenjamenMeyer/stackInABox ~ https://pypi.python.org/pypi/stackinabox https://github.com/BenjamenMeyer/openstackinabox ~ https://pypi.python.org/pypi/openstackinabox __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On 01/07/2016 03:01 PM, Jim Rollenhagen wrote: On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: Hi all, A change to global-requirements[1] introduces mimic, which is an http server that can mock various APIs, including nova and ironic, including control of error codes and timeouts. The ironic team plans to use this for testing python-ironicclient without standing up a full ironic environment. Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. // jim [1] https://review.openstack.org/#/c/220268/ What is the advantage of running another server like this over using requests-mock (which is used by other OpenStack projects for testing today)? The only difference here seems to be that you actually execute requests code in one case and not in the other. Requests-mock debugging when things go wrong seems a bit simpler. This is less about twisted and more about trying to not introduce yet another way to mock code in the tree that people need to understand. -Sean We'd be using this for functional tests, not unit, where we can't really inject mocks. The idea is that we could run a full functional suite against either mimic or a full ironic environment, just by changing a test setting. I don't really see the point of a separate project like Mimic that has a whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and RAX-specific) API services. It's just a great way to introduce a larger surface area for bugs to creep in -- since you have to keep the Mimic interfaces up to date with the real interfaces. Better to keep something like this -- if it is TRULY needed -- in-tree with the API service itself, so that the chances of divergence are reduced. This is similar to the fakevirt driver in Nova. It's in tree for good reason: when someone changes the virt driver interface, the fakevirt driver goes boom and needs to be changed in a corresponding fashion in the same patch. What value does a functional test against an HTTP API service that does nothing (other than introduce greater surface area for bugs) actually offer over unit tests anyway? -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote: > On 01/07/2016 02:09 PM, Jim Rollenhagen wrote: > > Hi all, > > > > A change to global-requirements[1] introduces mimic, which is an http > > server that can mock various APIs, including nova and ironic, including > > control of error codes and timeouts. The ironic team plans to use this > > for testing python-ironicclient without standing up a full ironic > > environment. > > > > Here's the catch - mimic is built on twisted. I know twisted was > > previously removed from OpenStack (or at least people said "pls no", I > > don't know the full history). We didn't intend to stealth-introduce > > twisted back into g-r, but it was pointed out to me that it may appear > > this way, so here I am letting everyone know. lifeless pointed out that > > when tests are failing, people may end up digging into mimic or twisted > > code, which most people in this community aren't familiar with AFAIK, > > which is a valid point though I hope it isn't required often. > > > > So, the primary question here is: do folks have a problem with adding > > twisted here? We're holding off on Ironic changes that depend on this > > until this discussion has happened, but aren't reverting the g-r change > > until we decide one way or another. > > > > // jim > > > > [1] https://review.openstack.org/#/c/220268/ > > What is the advantage of running another server like this over using > requests-mock (which is used by other OpenStack projects for testing > today)? The only difference here seems to be that you actually execute > requests code in one case and not in the other. > > Requests-mock debugging when things go wrong seems a bit simpler. > > This is less about twisted and more about trying to not introduce yet > another way to mock code in the tree that people need to understand. > > -Sean We'd be using this for functional tests, not unit, where we can't really inject mocks. The idea is that we could run a full functional suite against either mimic or a full ironic environment, just by changing a test setting. // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
It's also worth noting that the mimic team, along with other Rackers who work on Twistd, all worked to get python 3 support for mimic and associated dependencies in order to get this into OpenStack. I think it's safe to say this is a very friendly upstream and will help resolve any issues we might suss out. Thanks, Jay Faulkner From: Dmitry Tantsur <divius.ins...@gmail.com> Sent: Thursday, January 7, 2016 11:35 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all] re-introducing twisted to global-requirements 2016-01-07 20:09 GMT+01:00 Jim Rollenhagen <j...@jimrollenhagen.com<mailto:j...@jimrollenhagen.com>>: Hi all, A change to global-requirements[1] introduces mimic, which is an http server that can mock various APIs, including nova and ironic, including control of error codes and timeouts. The ironic team plans to use this for testing python-ironicclient without standing up a full ironic environment. Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. Btw, I've spent some amount of time (5 years?) with twisted on my previous jobs. While my memory is no longer fresh on it, I can definitely be pinged to help with it, if problems appear. So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. // jim [1] https://review.openstack.org/#/c/220268/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -- -- Dmitry Tantsur -- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
2016-01-07 20:09 GMT+01:00 Jim Rollenhagen: > Hi all, > > A change to global-requirements[1] introduces mimic, which is an http > server that can mock various APIs, including nova and ironic, including > control of error codes and timeouts. The ironic team plans to use this > for testing python-ironicclient without standing up a full ironic > environment. > > Here's the catch - mimic is built on twisted. I know twisted was > previously removed from OpenStack (or at least people said "pls no", I > don't know the full history). We didn't intend to stealth-introduce > twisted back into g-r, but it was pointed out to me that it may appear > this way, so here I am letting everyone know. lifeless pointed out that > when tests are failing, people may end up digging into mimic or twisted > code, which most people in this community aren't familiar with AFAIK, > which is a valid point though I hope it isn't required often. > Btw, I've spent some amount of time (5 years?) with twisted on my previous jobs. While my memory is no longer fresh on it, I can definitely be pinged to help with it, if problems appear. > > So, the primary question here is: do folks have a problem with adding > twisted here? We're holding off on Ironic changes that depend on this > until this discussion has happened, but aren't reverting the g-r change > until we decide one way or another. > > // jim > > [1] https://review.openstack.org/#/c/220268/ > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- -- -- Dmitry Tantsur -- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] re-introducing twisted to global-requirements
Hi all, A change to global-requirements[1] introduces mimic, which is an http server that can mock various APIs, including nova and ironic, including control of error codes and timeouts. The ironic team plans to use this for testing python-ironicclient without standing up a full ironic environment. Here's the catch - mimic is built on twisted. I know twisted was previously removed from OpenStack (or at least people said "pls no", I don't know the full history). We didn't intend to stealth-introduce twisted back into g-r, but it was pointed out to me that it may appear this way, so here I am letting everyone know. lifeless pointed out that when tests are failing, people may end up digging into mimic or twisted code, which most people in this community aren't familiar with AFAIK, which is a valid point though I hope it isn't required often. So, the primary question here is: do folks have a problem with adding twisted here? We're holding off on Ironic changes that depend on this until this discussion has happened, but aren't reverting the g-r change until we decide one way or another. // jim [1] https://review.openstack.org/#/c/220268/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] re-introducing twisted to global-requirements
A bit more information. the dependency for twisted is not in global-requirements.txt, it will be only added to upper-constraints.txt when the CI job/bot proposes it next. Thanks, Dims On Thu, Jan 7, 2016 at 2:09 PM, Jim Rollenhagenwrote: > Hi all, > > A change to global-requirements[1] introduces mimic, which is an http > server that can mock various APIs, including nova and ironic, including > control of error codes and timeouts. The ironic team plans to use this > for testing python-ironicclient without standing up a full ironic > environment. > > Here's the catch - mimic is built on twisted. I know twisted was > previously removed from OpenStack (or at least people said "pls no", I > don't know the full history). We didn't intend to stealth-introduce > twisted back into g-r, but it was pointed out to me that it may appear > this way, so here I am letting everyone know. lifeless pointed out that > when tests are failing, people may end up digging into mimic or twisted > code, which most people in this community aren't familiar with AFAIK, > which is a valid point though I hope it isn't required often. > > So, the primary question here is: do folks have a problem with adding > twisted here? We're holding off on Ironic changes that depend on this > until this discussion has happened, but aren't reverting the g-r change > until we decide one way or another. > > // jim > > [1] https://review.openstack.org/#/c/220268/ > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev