Dear TSC, I promised during today's TSC call to provide more detailed information on the proposed scope of the next OVP release (2018.0x):
* Slides showing a concise mapping of test suites to proposed categories and requirement levels (mandatory / optional) https://wiki.opnfv.org/download/attachments/2925933/Dovetail-OVP%20-%20TSC%20scope%20review.pdf?version=1&modificationDate=1530025340369&api=v2 * List of all proposed new test cases to be included in addition to the existing scope (OVP 2018.01) https://wiki.opnfv.org/display/dovetail/Proposed+Test+Scope+of+OVP+2018.0x * State of the proposed test cases in our CI https://wiki.opnfv.org/display/dovetail/Status+of+proposed+tests Please note that some of the failing test cases do in fact pass in the normal daily CI runs of the installers. The dovetail test suite is executed after Functest and Yardstick and we do see an irregular (PASS/FAIL) behavior across all platforms. I attribute this to the fact that the deployments have already gone through quite some test load which causes instabilities. If a test passes regularly in the daily installer runs and shows unstable behavior in the Dovetail runs, we still consider it mature enough for inclusion. * Bottlenecks ping stress test There was a request for more information on the bottlenecks stress test we are proposing for inclusion. In a nutshell, the test spawns a configurable number of threads, each deploying a Heat stack consisting of two VMs through Yardstick. The two VMs attempt to verify connectivity by means of pings. The test passes if the number of successfully deployed stacks is greater than a configurable threshold. In its current configuration, the test deploys 2x 20 stacks / threads and requires that all 20 stacks deploy successfully [1]. Unfortunately, we observe the testcase to fail regularly (observe the "result" and "success_rate" fields: [2]) given its current parameterization. We received the input on the call that Dovetail should not change the parameterization of test cases, but delegate this to the corresponding testing projects. [1] https://github.com/opnfv/bottlenecks/blob/master/testsuites/posca/testcase_cfg/posca_factor_ping.yaml [2] http://testresults.opnfv.org/test/api/v1/results?project=bottlenecks&case_name=posca_factor_ping Please review the proposal and come back with questions and remarks where needed. We are planning to review and approve the scope in one of the next TSC meetings. Best regards Georg From: Georg Kunz Sent: Thursday, June 14, 2018 10:43 AM To: opnfv-...@lists.opnfv.org Cc: xudan (N) <xuda...@huawei.com>; Cooper, Trevor <trevor.coo...@intel.com>; Lincoln Lavoie <lylav...@iol.unh.edu> Subject: [opnfv-tsc] Review of proposed scope for OVP 2018.0x Dear TSC, As part of my OVP release update during Tuesday's TSC call, I took an action item to compile a wiki page providing an overview of the new test cases under evaluation for the next OVP release as well as some pointers to in-depth information about the tests (documentation or code): https://wiki.opnfv.org/display/dovetail/Proposed+Test+Scope+of+OVP+2018.0x Please feel free to review and comment. The page might undergo updates as we are finalizing the scope. Best regards Georg -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21466): https://lists.opnfv.org/g/opnfv-tech-discuss/message/21466 Mute This Topic: https://lists.opnfv.org/mt/22685504/21656 Group Owner: opnfv-tech-discuss+ow...@lists.opnfv.org Unsubscribe: https://lists.opnfv.org/g/opnfv-tech-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
