Re: [openstack-dev] [Summit][qa] Vancouver Summit 2018 QA Recap

[Chandan kumar]

Hello Ghanshyam,

Thanks for putting this all together. Great summary :-)

On Fri, Jun 1, 2018 at 12:27 PM, Ghanshyam  wrote:
> Hi All,
>
> We had another good Summit in Vancouver  and got good amount of feedback for 
> QA which really important and helpful.
> I am summarizing the QA discussions during Summit.
>
> QA feedback sessions:
> =
> Etherpad: https://etherpad.openstack.org/p/YVR18-forum-qa-ops-user-feedback
> We had good number of people this time and so does more feedback.
>
> Key points, improvement and features requested in QA:
> - AT Cloud QA is by AQuA API which is tooling around upstream tools like 
> Tempest, Patrole, OpenStack Health etc.
> - Tempest, Patrole are widely used tool in Cloud testing. Patrole is being 
> used with 10 Roles in parallel testing on containers.
> - There are few more support needed from Tempest which AT (Doug 
> Schveninger) would like to see in upstream. Few of them are:
> - Better support for LDAP
> - Service available detection for plugins
> - Configure volume_type for Cinder multiple storage types tests
> - more tooling in Tempest like - tempest.conf generator,

For generating tempest.conf, we have python-tempestconf
 , It
might help.

> iproject_generator.py, advance cleanup/Leak detector,

> assembling tempest plugin in a docker container etc

By the beginning of Rocky cycle, we have added all tempest plugins in
Kolla tempest container and it is currently consumed in TripleO CI.
https://hub.docker.com/r/kolla/centos-source-tempest/tags/

It might help.

> - Tempest gabbi support
>
> ACTION ITEM:  gmann to follow up on each requested features and start 
> discussion in separate thread/IRC.
>
> Tagging all the Tempest plugins along with Tempest tag
> =
> Currently, we tag Tempest on release, intermediately or EOL  so that people 
> can use that tag against particular openstack code base/release.  Tempest 
> plugins are not being tagged as such.  So there are difficulty in using 
> plugins with particular Tempest tag in compatible way. We discussed to tag 
> all tempest plugins together everytime Tempest new tag is pushed. While 
> writing this mail, I got to know that dmellado already doing the new tag for 
> kuryr tempest plugin which is what we need.
>
> ACTION ITEM: gmann to start the ML thread to get the broader agreement from 
> each plugins and then define the process and responsible team to tag all 
> plugins and Tempest together.
>
> Patrole
> ==
> This is one of the important project now which is being requested/talked by 
> many people/operator. This was one the item in keystone Default Roles forum 
> session[1] also to start gating patrole on keystone. Below is initial plan I 
> discussed with Felipe:
> - Start gating patrole in keystone with non-voting/experimental job. This one 
> - https://review.openstack.org/#/c/464678/ . Rocky.
> - multi-policy support - Rocky
> - Make  stable release of Patrole. S cycle may be. This include various 
> things about framework stability, plugin support etc
> - Start proposing the Patrole gating on other projects like nova, cinder etc 
> - T Cycle or early if possible.
>
> ACTION ITEM: Felipe to work on above plan and gmann will be helping him on 
> that.
>
> QA onboarding sessions:
> ===
> Etherpad: https://etherpad.openstack.org/p/YVR18-forum-qa-onboarding-vancouver
>
> Around  6-7 people joined which gradually increasing since previous summits 
> :). We started with asking people about their engagement in QA or what they 
> are looking forward from QA.
> Doug Schveninger(AT) talked about his team members who can helps on QA 
> things and the new features/tooling he would like to see in Tempest, Patrole 
> etc. They might not be permanent but it is good to have more people in 
> contribution. QA team will help to get them on-boarded in all perspective. 
> Thanks Doug for your support.
>
> Other item fro this sessions was to have a centralized place (etherpad, 
> document) for all the current feature or working items where we are looking 
> for volunteer like CLI unit tests, schema validation etc. Where we document 
> the enough background and helping material which will help new contributors 
> to start working on those items.
>
> ACTION ITEM:
> - gmann to find the better place to document the working item with enough 
> background for new contributors.
> - Doug to start his team member to get involve in QA.
>
> Extended Maintenance Stable Branch
> =
> During discussion of Extended Maintenance sessions[2], we discussed about 
> testing support of EM branch in QA and we all agreed on below points:
> - QA will keep doing the same number of stable branches support as it is 
> doing now. Means support till "Maintained"  phase branches. EM branch will 
> not be in scope of guaranteed support of QA.
> - As Tempest is branchless, it 

[openstack-dev] [Summit][qa] Vancouver Summit 2018 QA Recap

[Ghanshyam]

Hi All,

We had another good Summit in Vancouver  and got good amount of feedback for QA 
which really important and helpful. 
I am summarizing the QA discussions during Summit. 

QA feedback sessions:
=
Etherpad: https://etherpad.openstack.org/p/YVR18-forum-qa-ops-user-feedback
We had good number of people this time and so does more feedback.

Key points, improvement and features requested in QA:
- AT Cloud QA is by AQuA API which is tooling around upstream tools like 
Tempest, Patrole, OpenStack Health etc.
- Tempest, Patrole are widely used tool in Cloud testing. Patrole is being used 
with 10 Roles in parallel testing on containers. 
- There are few more support needed from Tempest which AT (Doug Schveninger) 
would like to see in upstream. Few of them are:
- Better support for LDAP
- Service available detection for plugins
- Configure volume_type for Cinder multiple storage types tests
- more tooling in Tempest like - tempest.conf generator, 
iproject_generator.py, advance cleanup/Leak detector, assembling tempest plugin 
in a docker container etc
- Tempest gabbi support 

ACTION ITEM:  gmann to follow up on each requested features and start 
discussion in separate thread/IRC. 

Tagging all the Tempest plugins along with Tempest tag
=
Currently, we tag Tempest on release, intermediately or EOL  so that people can 
use that tag against particular openstack code base/release.  Tempest plugins 
are not being tagged as such.  So there are difficulty in using plugins with 
particular Tempest tag in compatible way. We discussed to tag all tempest 
plugins together everytime Tempest new tag is pushed. While writing this mail, 
I got to know that dmellado already doing the new tag for kuryr tempest plugin 
which is what we need. 

ACTION ITEM: gmann to start the ML thread to get the broader agreement from 
each plugins and then define the process and responsible team to tag all 
plugins and Tempest together. 

Patrole
==
This is one of the important project now which is being requested/talked by 
many people/operator. This was one the item in keystone Default Roles forum 
session[1] also to start gating patrole on keystone. Below is initial plan I 
discussed with Felipe: 
- Start gating patrole in keystone with non-voting/experimental job. This one - 
https://review.openstack.org/#/c/464678/ . Rocky. 
- multi-policy support - Rocky
- Make  stable release of Patrole. S cycle may be. This include various things 
about framework stability, plugin support etc
- Start proposing the Patrole gating on other projects like nova, cinder etc - 
T Cycle or early if possible. 

ACTION ITEM: Felipe to work on above plan and gmann will be helping him on 
that. 

QA onboarding sessions:
===
Etherpad: https://etherpad.openstack.org/p/YVR18-forum-qa-onboarding-vancouver 

Around  6-7 people joined which gradually increasing since previous summits :). 
We started with asking people about their engagement in QA or what they are 
looking forward from QA. 
Doug Schveninger(AT) talked about his team members who can helps on QA things 
and the new features/tooling he would like to see in Tempest, Patrole etc. They 
might not be permanent but it is good to have more people in contribution. QA 
team will help to get them on-boarded in all perspective. Thanks Doug for your 
support. 

Other item fro this sessions was to have a centralized place (etherpad, 
document) for all the current feature or working items where we are looking for 
volunteer like CLI unit tests, schema validation etc. Where we document the 
enough background and helping material which will help new contributors to 
start working on those items. 

ACTION ITEM: 
- gmann to find the better place to document the working item with enough 
background for new contributors. 
- Doug to start his team member to get involve in QA.

Extended Maintenance Stable Branch
=
During discussion of Extended Maintenance sessions[2], we discussed about 
testing support of EM branch in QA and we all agreed on below points:
- QA will keep doing the same number of stable branches support as it is doing 
now. Means support till "Maintained"  phase branches. EM branch will not be in 
scope of guaranteed support of QA. 
- As Tempest is branchless, it should work for EM phase branches also but if 
anything new changes break EM branch testing then we stopped testing master 
Tempest on EM branches. 
Matt has already pushed the patch to document the above agreement [3]. Thanks 
for doing good documentation always :), 

Eris
===
Spec- https://review.openstack.org/#/c/443504/
It came up in feedback sessions also and people really want to see some 
progress on this. We have spec under review for that and need more volunteer to 
drive this forward. I will also check with SamP on this. Other than that there 
was not much discussion/progress on this in summit.

ACTION ITEM:  gmann