Hello, I fully support too.
For a) It should be noted that we have developed lots of unit tests to fully cover our Framework (the global coverage is quite good too). We are checking our code via PEP8 [1] and Pylint [2] to increase continuously the code quality. For instance, Framework must be rated 10/10 via Pylint. Otherwise any proposal is automatically refused (Jenkins -1). We also forbid any unix permission mistake via tox [3]. We can proudly write that we are applying the golden rules of OpenStack. And more we try to propagate them out of Functest (see requirements [3]) I don’t think all OPNFV projects have the same level of quality simply because our request to install Violations plugin (Jenkins [4]) was refused. [1] https://www.python.org/dev/peps/pep-0008/ [2] https://www.pylint.org/ [3] https://wiki.opnfv.org/display/functest/Requirements+management [4] https://wiki.jenkins.io/display/JENKINS/Violations Cédric De : opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] De la part de morgan.richo...@orange.com Envoyé : vendredi 20 octobre 2017 10:36 À : David McBride; Jose Lausuch Cc : TSC OPNFV; TECH-DISCUSS OPNFV Objet : Re: [opnfv-tech-discuss] [opnfv-tsc] [release][euphrates] recommended list of scenarios for release in Euphrates 5.0 Hi, I fully support Jose's comments "I've been told (a) the test frameworks are not sufficiently mature to be used for gating release; and (b) the community prefers to leave the decision about quality to the scenario owners. So, we are left with (1) deploy status; and (2) scenario owner judgment for determining scenario release." for b) I think the scenario owner may decide what kind of tests he/she wants to run on his/her scenario to highlight the feature/performance/ ... no need to run a vIMS onboarding if your scenario is focusing on data plane acceleration However basic tests (for functest healthcheck & smoke) must be PASS to validate the scenario, it gives a minimum trust into the scenario... Without a minimum Trust, I do not see how we could justify any certification program. It is like a constitution, the power shall be balanced...it is not up to the scenario owner to declare and validate a scenario. As release manager, you should be the supreme court and guarantee this separation of powers.. for a) could you just elaborate a little bit.... As far as I know I never got this feedback, so who told you that? what are the rationals behind that? what kind of maturity level do you expect? Functest is ready since the first official release date - 2 weeks ago Some scenarios already reached the criteria several weeks before the first release date including VNF onboarding testing (which are not part of smoke and healthcheck) the framework is flexible and allows black listing of tests if scenario owner can justify any exception (upstream bugs, configuration restrictions,..) The healthcheck tier is very stable since MS3, I do not think the gate has been broken once for euphrates.. I remember you send a mail to encourage testing projects for the integration phase It is a bit surprising to see at the end that the release criteria is almost declarative /Morgan On 19/10/2017 17:41, David McBride wrote: Hi Jose, Scenario owners express intent-to-release on the scenario status page<https://wiki.opnfv.org/display/SWREL/Euphrates+Scenario+Status>. Scenario owners indicated that they wanted to release each scenario in the recommended list. I would very much like to use test results to gate release of scenarios. That would be my preference. However, each time this issue has come up, I've been told (a) the test frameworks are not sufficiently mature to be used for gating release; and (b) the community prefers to leave the decision about quality to the scenario owners. So, we are left with (1) deploy status; and (2) scenario owner judgment for determining scenario release. We can certainly change this for future releases. In order to do that, I'd like to have a recommendation on test requirements from the test working group, including which tests are gating, minimum test runs, the number of consecutive iterations that must pass, etc. Once we have approval from the TSC, then we can put those requirements in place. David On Thu, Oct 19, 2017 at 8:01 AM, Jose Lausuch <jalaus...@suse.com<mailto:jalaus...@suse.com>> wrote: Hi, I just want to raise a concern the Functest team has been discussing this morning. This list is maybe good from a deployment prospective, but it seems we are lowering the bar the older OPNFV gets… I remember in the Arno and Brahmaputra days the release criteria was to pass Functest/Yardstick, then we said that scenario owners can decide to release or not release according to the results. Now it seems that a deployable scenario is eligible to be released without taking into account the quality, which can be acceptable.. but the community and the users should know that some of these scenarios do not even pass Healthcheck or vPing whereas others get 100% success from Functest. I am not saying we should not release them but if I were an user, I would like to know some details about the quality of the scenarios. I would not like to deploy something that cannot even spawn VMs or do a simple ping between them… Maybe for the next time, we could have a general release document/report about the quality of each scenario and mention the known issues for those which have problems to pass the minimum set of tests. Functest is doing this by taking a snapshot of the dashboard at the release date [1], but I do not think we should be the project responsible for offering that information, it should be more visible and centralized. Regards, Jose [1] http://testresults.opnfv.org/functest/euphrates/ On 17 Oct 2017, at 22:50, David McBride <dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>> wrote: Team, Based on scenario owner intent-to-participate, as well as a review of deploy results, I'm recommending that the following scenarios be released for Euphrates 5.0: Scenario Installer os-nosdn-nofeature-ha Apex os-nosdn-calipso-noha Apex os-nosdn-nofeature-noha Apex os-odl-nofeature-ha Apex os-odl-nofeature-noha Apex os-nosdn-ovs_dpdk-ha Apex os-nosdn-ovs_dpdk-noha Apex os-odl-fdio-noha Apex os-odl-fdio-ha Apex os-odl-fdio_dvr-noha Apex os-nosdn-fdio-noha Apex os-nosdn-fdio-ha Apex os-nosdn-kvm_ovs_dpdk-ha Apex os-nosdn-kvm_ovs_dpdk-noha Apex os-nosdn-bar-ha Apex os-nosdn-bar-noha Apex os-odl-bgpvpn-ha Apex os-odl-bgpvpn-noha Apex os-odl-sfc-noha Apex os-ovn-nofeature-noha Apex os-odl_l2-moon-ha Compass os-odl_l2-moon-noha Compass os-nosdn-nofeature-ha Compass4NFV os-nosdn-nofeature-noha Compass4NFV os-odl-sfc-ha Compass4NFV os-odl-sfc-noha Compass4NFV k8s-nosdn-nofeature-ha Compass4NFV os-nosdn-nofeature-ha Daisy os-odl-nofeature-ha Daisy os-nosdn-nofeature-noha Fuel/MCP os-nosdn-nofeature-ha Fuel/MCP os-nosdn-ovs-noha Fuel/MCP os-nosdn-ovs-ha Fuel/MCP os-odl-nofeature-noha Fuel/MCP os-odl-nofeature-ha Fuel/MCP k8-nosdn-lb-noha JOID os-nosdn-nofeature-ha JOID os-nosdn-lxd-ha JOID os-nosdn-lxd-noha JOID os-nosdn-nofeature-noha JOID os-ocl-nofeature-ha JOID os-ocl-nofeature-noha JOID os-nosdn-openbaton-ha JOID k8-ovn-lb-noha JOID os-nosdn-nofeature-ha Fuel/MCP (aarch64) os-odl-nofeature-ha Fuel/MCP (aarch64) -- David McBride Release Manager, OPNFV Mobile: +1.805.276.8018<tel:%2B1.805.276.8018> Email/Google Talk: dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org> Skype: davidjmcbride1 IRC: dmcbride _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss -- David McBride Release Manager, OPNFV Mobile: +1.805.276.8018<tel:%2B1.805.276.8018> Email/Google Talk: dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org> Skype: davidjmcbride1 IRC: dmcbride _______________________________________________ opnfv-tsc mailing list opnfv-...@lists.opnfv.org<mailto:opnfv-...@lists.opnfv.org> https://lists.opnfv.org/mailman/listinfo/opnfv-tsc -- Morgan Richomme Orange/ IMT/ OLN/ CNC/ NCA/ SINA Network architect for innovative services Future of the Network community member Open source Orange community manager tel. +33 (0) 296 072 106 mob. +33 (0) 637 753 326 morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
