Dear Bats, thanks everyone for participating to the Queens PTG - in person and remotely - and for making it a successful and enjoyable event.
Here's a summary of what we discussed and achieved during the week. This report is far from complete - please complement it with QA related stories I may have missed. Unfortunately I ran out of bat stickers, I hope to have some more for the next time we meet :) Pike Retrospective ------------------------- We experimented this time with running a retrospective [0]. The input for the retrospective was almost exclusively from members of the QA team, which means that either the retrospective was poorly advertised by me, or that we did a nice job for the community during the QA cycle. I hope the latter :) The outcome of the retrospective was mostly 'yay' for tasks completed, and 'next cycle' for things we did not have time to work on. A few extra things that came up were: - bug triage and fix: we may need a bug czar and more automation to stay on top of the bug queue - elastic recheck categorisation: we may need an e-r czar to ensure we don't leave gate failures and races uncategorised and accumulating - meetings are mostly attended in the APAC time zone and very seldom attended by non-qa folks. We will consider shortening / reducing them and complementing them with QA office hours Monitoring the Gate -------------------------- At the beginning of the Pike cycle we had a number of stability issues in the gate. To prevent issues from accumulating over time we discussed a few ideas about monitoring the status of the gate and providing more feedback to reviewers to help catch patches that may introduce issues [1]. There are a few things that can be done relatively easily (or that we already have) in terms of data collection: job duration and failure rates, aggregated dstat data, resources created by tests. We miss OpenStack Health contributors to create new views for this data. Links from gerrit and zuul dashboard into OpenStack Health would help making the data discoverable. Tempest ------------ A large chunk of the patches required to mark test.py as a stable interface for plugins was merged during the summit. It was good to have them merged during PTG so it was easier to fix resulting issues in Tempest plugins - at least Manila and Sahara needed a small patch. There are two patch series left [2][3] which should have very little impact on plugins - we'll try to merge them as soon as possible to avoid disruptions later in the Queens cycle. We worked with a few project teams on the goal to split Tempest plugins to a dedicated repo. The step by step process [4] linked to the goal includes an example section - we hope to get more and more examples in there from team who already went through the process. Devstack ------------ We discussed about devstack runtime. The time to setup an OpenStack cloud using devstack seems to have increased over time [5][6] - it would be good to investigate why and see if there is time we can save in the gate and on each developer laptop :) Between August and September the average runtime in the gate increased by about 200s. Upgrade Testing ---------------------- Rolling upgrade testing via grenade is important for project that seek obtaining the support rolling upgrade tag [7]. While the scope of Grenade is rather fixed, it should be possible to support ordering (or relative ordering) in project updates. Policy Testing ------------------ We discussed what's next for the Queens cycle - support for multi-policy testing is the largest chunk of work planned for now. stestr ------- The migration in ostestr to use stestr internally happened shortly before the PTG [8] and we worked through the PTG to amend any deviation in behaviour that this may have caused. Next in the todo list is to run stestr natively in Tempest, bypassing ostestr completely. The plan is for this to lead the way for projects to gradually remove the dependency to ostestr completely. HA Testing --------------- We talked quite a bit about HA and non-functional testing in general. Non-functional testing is not a good fit for gate testing, since it's not as reliable as functional / integration testing and it often produces results which needs to be interpreted by a human being. It also has strong dependencies to the deployment tooling and architecture. Until now most of OpenStack non-functional testing has been done by vendors and operators using downstream tools. SamP and guatamdivgi presented to the QA team a proposal for an Ansible based framework for HA testing [9]. Plugins will allow to seamlessly port different test scenario against different cloud architecture, thus rendering the framework of interest as a general testing tool for OpenStack. The same concept can be extended to non-functional testing in general. It's not clear yet if any of this could run as part of OpenStack CI. We hope to see a PoC after a couple of months in the Queens cycle. The NFV ecosystem would be happy to see publicly available non-functional tests for OpenStack as well - if we had a common framework for such tests we'd have an opportunity to attract contributions into tests from them as well. All things Zuul V3 ----------------------- Zuul v3 will provide a DB backend for test runs data, which we want to integrate as a new data source for OpenStack Health, so that we can then provide a full picture of all job runs in O-H. Job names will change with zuul v3. While we found a simple solution to keep OpenStack Health data functional with Zuul v3 provided metadata, we will still loose 6 months of historical data in the process of migration, for which we don't have a solution yet. In terms of Zuul v3 jobs, we started working on Zuul v3 native Tempest jobs [10]. this will be the basis for most jobs that run Tempest against devstack. The patch is still WIP, but it already runs the full Tempest run, with only one test failure which I need to investigate. QA Help Room -------------------- The help room was helpful (heh, I couldn't resist) mostly on day 1. It was good to have dedicated time allocated to answer questions, since it gave us space to open the laptop and work on code directly to answer specific questions. There were no questions asked during day 2. We used the time to join other sessions and work on QA topics, but perhaps in Dublin we could consider having a single help room day. Cross-Community ------------------------ NFV: It was very interesting to discuss with the OPNFV community about CI and test processes and architecture. The aim on both sides to re-use / share tools and best practices, share test results and avoid duplicate efforts where relevant. K8S: We discussed with dmellado the possibility of having a basic k8s client hosted in a Tempest plugin [11], that could be used by k8s related OpenStack project to write OpenStack / k8s scenario tests. The main advantages of a k8s client written in Tempest format and hosted by OpenStack would be a uniform interface with the OpenStack Tempest client and a stable testing API. QA Themes for Queens -------------------------------- We summarised the topics the QA team will focus on through the Queens cycle in out queens priority etherpad [11]. The main etherpad for QA at the PTG in Denver is available as well [12]. QA Team Pictures ------------------------ I'll forward the pictures as soon as I have a high-res copy available :) Thanks for reading! Andrea Frittoli (andreaf) [0] https://etherpad.openstack.org/p/qa-pike-retrospective [1] https://etherpad.openstack.org/p/qa-pike-gate-performance [2] https://review.openstack.org/#/q/topic:test_module_stable+(status:open+OR+status:merged ) [3] https://review.openstack.org/#/q/topic:bp/consistent-service-method-names+status:open+owner:%22Ghanshyam+Mann+%253Cghanshyammann%2540gmail.com%253E%22 [4] https://etherpad.openstack.org/p/tempest-separate-plugin [5] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122271.html [6] http://status.openstack.org/openstack-health/#/test/devstack?resolutionKey=day&duration=P6M [7] https://governance.openstack.org/tc/reference/tags/assert_supports-rolling-upgrade.html [8] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122135.html [9] https://etherpad.openstack.org/p/qa-queens-ptg-destructive-testing [10] https://review.openstack.org/#/c/504246/ [11] https://etherpad.openstack.org/p/qa-queens-priorities [12] https://etherpad.openstack.org/p/qa-queens-ptg
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
