On 05/01/2014 01:30 PM, David Kranz wrote: > On 05/01/2014 11:36 AM, Matthew Treinish wrote: >> On Thu, May 01, 2014 at 06:18:10PM +0900, Ken'ichi Ohmichi wrote: >>> # Sorry for sending this again, previous mail was unreadable. >>> >>> 2014-04-28 11:54 GMT+09:00 Ken'ichi Ohmichi <[email protected]>: >>>>> This is also why there are a bunch of nova v2 extensions that just add >>>>> properties to an existing API. I think in v3 the proposal was to do >>>>> this with >>>>> microversioning of the plugins. (we don't have a way to configure >>>>> microversioned v3 api plugins in tempest yet, but we can cross that >>>>> bridge when >>>>> the time comes) Either way it will allow tempest to have in config >>>>> which >>>>> behavior to expect. >>>> Good point, my current understanding is: >>>> When adding new API parameters to the existing APIs, these >>>> parameters should >>>> be API extensions according to the above guidelines. So we have >>>> three options >>>> for handling API extensions in Tempest: >>>> >>>> 1. Consider them as optional, and cannot block the incompatible >>>> changes of them. (Current) >>>> 2. Consider them as required based on tempest.conf, and can block the >>>> incompatible changes. >>>> 3. Consider them as required automatically with microversioning, and >>>> can block the incompatible changes. >>> I investigated the way of the above option 3, then have one question >>> about current Tempest implementation. >>> >>> Now verify_tempest_config tool gets API extension list from each >>> service including Nova and verifies API extension config of tempest.conf >>> based on the list. >>> Can we use the list for selecting what extension tests run instead of >>> the verification? >>> As you said In the previous IRC meeting, current API tests will be >>> skipped if the test which is decorated with requires_ext() and the >>> extension is not specified in tempest.conf. I feel it would be nice >>> that Tempest gets API extension list and selects API tests automatically >>> based on the list. >> So we used to do this type of autodiscovery in tempest, but we stopped >> because >> it let bugs slip through the gate. This topic has come up several >> times in the >> past, most recently in discussing reorganizing the config file. [1] >> This is why >> we put [2] in the tempest README. I agree autodiscovery would be >> simpler, but >> the problem is because we use tempest as the gate if there was a bug >> that caused >> autodiscovery to be different from what was expected the tests would just >> silently skip. This would often go unnoticed because of the sheer >> volume of >> tempest tests.(I think we're currently at ~2300) I also feel that >> explicitly >> defining what is a expected to be enabled is a key requirement for >> branchless >> tempest for the same reason. > >> >> The verify_tempest_config tool was an attempt at a compromise between >> being >> explicit and also using auto discovery. By using the APIs to help >> create a >> config file that reflected the current configuration state of the >> services. It's >> still a WIP though, and it's really just meant to be a user tool. I >> don't ever >> see it being included in our gate workflow. > I think we have to accept that there are two legitimate use cases for > tempest configuration: > > 1. The entity configuring tempest is the same as the entity that > deployed. This is the gate case. > > 2. Tempest is to be pointed at an existing cloud but was not part of a > deployment process. We want to run the tests for the supported > services/extensions. > > We should modularize the code around discovery so that the discovery > functions return the changes to conf that would have to be made. The > callers can then decide how that information is to be used. This would > support both use cases. I have some changes to the verify_tempest_config > code that does this which I will push up if the concept is agreed.
Discovery is a separate thing from testing. We've seen the discovery
issue go wrong and return "everything passed" when *much less* than you
expected to be run was run.
Matt's got some tooling to build some of that out of band, which is good.
But this is a part where adding friction here is good user experience.
Because user experience is not only about making it easy to run things
correctly, but making it *hard* to run them incorrectly.
-Sean
--
Sean Dague
http://dague.net
signature.asc
Description: OpenPGP digital signature
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
