Looking at the tool, it seems to me that is servers a combination of functions: - provision test resources - support for distribution specific / cloud specific overrides to default - support for configuration override via CLI - discovery of configuration
Test resource provisioning is something that I agree is useful to have. We plan in Mitaka to separate out of tempest.conf the configuration of test resources, and have a CLI tool to provision them [0]. We could re-use code from this tool to achieve that. Support for distribution specific / cloud specific overrides is also something that is useful. In clouds where I control the deployment process I inject extra configs in tempest.conf based on the deployment options. In clouds where I don't, I maintain a partial tempest.conf with the list of options which I expects I have to modify to match the target cloud. That's pretty easy to achieve though - simply append the extra configs to the bottom of tempest.conf and it's done. Duplicate configuration options are not an issue, the last one wins. Still we could support specifying a number of configuration files to the non-yet implemented "tempest run" command. Support for configuration override via CLI is something that I can see it can be useful during development or troubleshooting, we could support that as options of the non-yet implemented "tempest run" command. The last point is discovery - I believe we should use that only as we use it today in the gate - i.e. fail fast if the generated configuration does not match what can be discovered from the cloud. So I would not get the script as is into tempest, but I think many of the functions implemented by it can fit into tempest - and some are already there. andrea [0] https://review.openstack.org/#/c/173334/ On Mon, Nov 30, 2015 at 7:39 AM Masayuki Igawa <masayuki.ig...@gmail.com> wrote: > Hi, > > I agree with Ken'ichi's opinion, basically. Tempest users should know > "what do we test for?" and we shouldn't discover values that we test for > automatically. > If users seems that "My current cloud is good. This is what I expect.", > discovering function could work. But I suppose many of users would use > tempest-config-generator for a new their cloud. So I feel the tool users > could be misunderstanding easily. > > But I also think that tempest users don't need to know all of the > configurations. > So, how about something like introducing "a configuration wizard" for > tempest configuration? > This is a just idea, though.. > > > Anyway, if you have the motivation to introduce tempest-config, how about > writing a spec for the feature for a concrete discussion? > (I think we should have agreements about the target issues, users, > solutions, etc.) > > Best Regards, > -- Masayuki Igawa > > > 2015年11月29日(日) 22:07 Yair Fried <yfr...@redhat.com>: > >> Hi, >> I agree with Jordan. >> We don't have to use the tool as part of the gate. It's target audience >> is people and not CI systems. More specifically - new users. >> However, we could add a gate (or a few) for the tool that makes sure a >> proper conf file is generated. It doesn't have to run the tests, just >> compare the output of the script to the conf generated by devstack. >> >> Re Rally - I believe the best place for tempest configuration script is >> within tempest. That said, if the Tempest community doesn't want this tool, >> we'll have to settle for the Rally solution. >> >> Regards >> Yair >> >> On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier < >> jordan.pitt...@scality.com> wrote: >> >>> Hi, >>> I think this script is valuable to some users: Rally and Red Hat >>> expressed their needs, they seem clear. >>> >>> This tool is far from bullet proof and if used blindly or in case of >>> bugs, Tempest could be misconfigured. So, we could have this tool inside >>> the Tempest repository (in the tools/) but not use it at all for the Gate. >>> >>> I am not sure I fully understand the resistance for this, if we don"t >>> use this config generator for the gate, what's the risk ? >>> >>> Jordan >>> >>> On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com >>> > wrote: >>> >>>> 2015-11-27 15:40 GMT+09:00 Daniel Mellado <daniel.mellado...@ieee.org>: >>>> > I still do think that even if there are some issues addressed to the >>>> > feature, such as skipping tests in the gate, the feature itself it's >>>> still >>>> > good -we just won't use it for the gates- >>>> > Instead it'd be used as a wrapper for a user who would be interested >>>> on >>>> > trying it against a real/reals clouds. >>>> > >>>> > Ken, do you really think a tempest user should know all tempest >>>> options? >>>> > As you pointed out there are quite a few of them and even if they >>>> should at >>>> > least know their environment, this script would set a minimum >>>> acceptable >>>> > default. Do you think PTL and Pre-PTL concerns that we spoke of would >>>> still >>>> > apply to that scenario? >>>> >>>> If Tempest users run part of tests of Tempest, they need to know the >>>> options which are used with these tests only. >>>> For example, current Tempest contains ironic API tests and the >>>> corresponding options. >>>> If users don't want to run these tests because the cloud don't support >>>> ironic API, they don't need to know/setup these options. >>>> I feel users need to know necessary options which are used on tests >>>> they want, because they need to investigate the reason if facing a >>>> problem during Tempest tests. >>>> >>>> Now Tempest options contain their default values, but you need a >>>> script for changing them from the default. >>>> Don't these default values work for your cloud at all? >>>> If so, these values should be changed to better. >>>> >>>> Thanks >>>> Ken Ohmichi >>>> >>>> --- >>>> >>>> > Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it >>>> to >>>> > tempest-cli improvements? What do you think about this, Masayuki? >>>> > >>>> > Thanks for all your feedback! ;) >>>> > >>>> > El 27/11/15 a las 00:15, Andrey Kurilin escribió: >>>> > >>>> > Sorry for wrong numbers. The bug-fix for issue with counters is >>>> merged. >>>> > Correct numbers(latest result from rally's gate[1]): >>>> > - total number of executed tests: 1689 >>>> > - success: 1155 >>>> > - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] >>>> should >>>> > enable them) >>>> > - failed: 0 >>>> > >>>> > [1] - >>>> > >>>> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz >>>> > [2] - https://review.openstack.org/#/c/250540/ >>>> > >>>> > On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov < >>>> yloban...@mirantis.com> >>>> > wrote: >>>> >> >>>> >> Hello everyone, >>>> >> >>>> >> Yes, I am working on this now. We have some success already, but >>>> there is >>>> >> a lot of work to do. Of course, some things don't work ideally. For >>>> example, >>>> >> in [2] from the previous letter we have not 24 skipped tests, >>>> actually much >>>> >> more. So we have a bug somewhere :) >>>> >> >>>> >> Regards, >>>> >> Yaroslav Lobankov. >>>> >> >>>> >> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin < >>>> akuri...@mirantis.com> >>>> >> wrote: >>>> >>> >>>> >>> Hi! >>>> >>> Boris P. and I tried to push a spec[1] for automation tempest config >>>> >>> generator, but we did not succeed to merge it. Imo, qa-team doesn't >>>> want to >>>> >>> have such tool:( >>>> >>> >>>> >>> >However, there is a big concern: >>>> >>> >If the script contain a bug and creates the configuration which >>>> makes >>>> >>> >most tests skipped, we cannot do enough tests on the gate. >>>> >>> >Tempest contains 1432 tests and difficult to detect which tests are >>>> >>> >skipped as unexpected. >>>> >>> >>>> >>> Yaroslav Lobankov is working on improvement for tempest config >>>> generator >>>> >>> in Rally. Last time when we launch full tempest run[2], we got 1154 >>>> success >>>> >>> tests and only 24 skipped. Also, there is a patch, which adds x-fail >>>> >>> mechanism(it based on subunit-filter): you can transmit a file with >>>> test >>>> >>> names + reasons and rally will modify results. >>>> >>> >>>> >>> [1] - https://review.openstack.org/#/c/94473/ >>>> >>> >>>> >>> [2] - >>>> >>> >>>> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz >>>> >>> >>>> >>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi < >>>> ken1ohmi...@gmail.com> >>>> >>> wrote: >>>> >>>> >>>> >>>> Hi Daniel, >>>> >>>> >>>> >>>> Thanks for pointing this up. >>>> >>>> >>>> >>>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado < >>>> daniel.mellado...@ieee.org>: >>>> >>>> > Hi All, >>>> >>>> > >>>> >>>> > As you might already know, within Red Hat's tempest fork, we do >>>> have >>>> >>>> > one >>>> >>>> > tempest configuration script which was built in the past by David >>>> >>>> > Kranz [1] >>>> >>>> > and that's been actively used in our CI system. Regarding this >>>> topic, >>>> >>>> > I'm >>>> >>>> > aware that quite some effort has been done in the past [2] and I >>>> would >>>> >>>> > like >>>> >>>> > to complete the implementation of this blueprint/spec. >>>> >>>> > >>>> >>>> > My plan would be to have this script under the /tempest/cmd or >>>> >>>> > /tempest/tools folder from tempest so it can be used to >>>> configure not >>>> >>>> > the >>>> >>>> > tempest gate but any cloud we'd like to run tempest against. >>>> >>>> > >>>> >>>> > Adding the configuration script was discussed briefly at the >>>> Mitaka >>>> >>>> > summit >>>> >>>> > in the QA Priorities meting [3]. I propose we use the existing >>>> >>>> > etherpad to >>>> >>>> > continue the discussion around and tracking of implementing >>>> "tempest >>>> >>>> > config-create" using the downstream config script as a starting >>>> point. >>>> >>>> > [4] >>>> >>>> > >>>> >>>> > If you have any questions, comments or opinion, please let me >>>> know. >>>> >>>> >>>> >>>> This topic have happened several times, and I also felt this kind >>>> of >>>> >>>> tool was very useful for Tempest users, because Tempest contains >>>> 296 >>>> >>>> options($ grep cfg * -R | grep Opt | wc -l) now and it is >>>> difficult to >>>> >>>> set the configuration up. >>>> >>>> However, there is a big concern: >>>> >>>> If the script contain a bug and creates the configuration which >>>> makes >>>> >>>> most tests skipped, we cannot do enough tests on the gate. >>>> >>>> Tempest contains 1432 tests and difficult to detect which tests are >>>> >>>> skipped as unexpected. >>>> >>>> Actually we faced unexpected skipped tests on the gate before due >>>> to >>>> >>>> some bug, then the problem has been fixed. >>>> >>>> But I can imagine this kind of problem happens after implementing >>>> this >>>> >>>> kind of script. >>>> >>>> >>>> >>>> So now I am feeling Tempest users need to know what cloud they >>>> want to >>>> >>>> test with Tempest, and need to know what tests run with Tempest. >>>> >>>> Testers need to know what test target/items they are testing >>>> basically. >>>> >>>> >>>> >>>> Thanks >>>> >>>> Ken Ohmichi >>>> >>>> >>>> >>>> --- >>>> >>>> >>>> >>>> > --- >>>> >>>> > [1] >>>> >>>> > >>>> >>>> > >>>> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py >>>> >>>> > [2] >>>> >>>> > >>>> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator >>>> >>>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities >>>> >>>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> __________________________________________________________________________ >>>> >>>> > OpenStack Development Mailing List (not for usage questions) >>>> >>>> > Unsubscribe: >>>> >>>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> >>>> > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> > >>>> >>>> >>>> >>>> >>>> >>>> >>>> __________________________________________________________________________ >>>> >>>> OpenStack Development Mailing List (not for usage questions) >>>> >>>> Unsubscribe: >>>> >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> -- >>>> >>> Best regards, >>>> >>> Andrey Kurilin. >>>> >>> >>>> >>> >>>> >>> >>>> __________________________________________________________________________ >>>> >>> OpenStack Development Mailing List (not for usage questions) >>>> >>> Unsubscribe: >>>> >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>>> >> >>>> >> >>>> >> >>>> __________________________________________________________________________ >>>> >> OpenStack Development Mailing List (not for usage questions) >>>> >> Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Best regards, >>>> > Andrey Kurilin. >>>> > >>>> > >>>> > >>>> __________________________________________________________________________ >>>> > OpenStack Development Mailing List (not for usage questions) >>>> > Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> > >>>> > >>>> > >>>> > >>>> __________________________________________________________________________ >>>> > OpenStack Development Mailing List (not for usage questions) >>>> > Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> > >>>> >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > -- > -- Masayuki Igawa > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev