Hi Leo,
Thanks for getting the ball rolling on this. I don’t want to hack at the wiki, so you get this inline… The DoveTail suites should be built around validation areas which each have a suite of test cases associated with them. These test suites should be broken down into a number of test cases that test (and prove) the ability of an implementation to provide functionality necessary for the operation of an OPNFV platform. As such I would think out test report would be structured around the test results per test area (by test suite) with a summary of the results of the test suite. The results of a test suite should not result in the tool providing any statement of compliance or not to OPNFV at this time. The process we will follow is that the CVP will judge if an implementation is compliant based on the overall assessment of the results. Thus I propose: · Call it the “dovetail report” for now · we remove the line around ‘Compliance:’. · Add a link to the test dashboard. · We remove “create date” as the test may be run over a period of time. · Replace “Certify ID” with “Validation ID” Maybe something like: ---------------------------------------------------------------------------- DoveTail Report Version: 1.0 TestSuite: basic Result dashboard: <link to test result dashboard> Validation ID: 9b2a4144-21f8-7c2a-987b-8f2c125c2df2 Upload Date: 2016-11-01 20:27:21 UTC Duration: 514 seconds Assessed test areas: -vimoperations results: <link to document section with details> pass % </link > -networkoperations results: <link to document section with details> pass % </link > -networkipv6 results: <link to document section with details> pass % </link > -networkvpn results: <link to document section with details> pass % </link > -networkbgp results: <link to document section with details> pass % </link > etc… vimoperations: Pass rate % - <link to spec>dovetail.test.identifier1 </link >: result <link to test logs> - <link to spec>dovetail.test.identifier2 </link >: result <link to test logs> - <link to spec>dovetail.test.identifier3 </link >: result <link to test logs> networkoperations: Pass rate % - <link to spec>dovetail.test.identifier1 </link >: result <link to test logs> - <link to spec>dovetail.test.identifier2 </link >: result <link to test logs> - <link to spec>dovetail.test.identifier3 </link >: result <link to test logs> networkipv6: Pass rate % - <link to spec>dovetail.test.identifier1 </link >: result <link to test logs> - <link to spec>dovetail.test.identifier2 </link >: result <link to test logs> - <link to spec>dovetail.test.identifier3 </link >: result <link to test logs> Something like that I think. We should not in any way link to any tempest test cases in the test report, the test is defined by the document outlining the test cases. This can include a link to the dovetail.testcase.py scripts used, but not link outside of OPNFV. If someone was to do this manually they need to be able to provide manually entered results. Regards, Chris From: <[email protected]> on behalf of Leo Wang <[email protected]> Date: Tuesday, 15 November 2016 at 09:34 To: TECH-DISCUSS OPNFV <[email protected]> Subject: [opnfv-tech-discuss] [dovetail] candidate of report content Hi all With the project moving on, the report of dovetail also needs to evolve with more information to present the status of the test. Here is a candidate report style: ---------------------------------------------------------------------------- Compliance Report Version: 1.0 TestSuite: basic Certify ID: 9b2a4144-21f8-7c2a-987b-8f2c125c2df2 Create Date: 2016-11-01 20:17:21 UTC Upload Date: 2016-11-01 20:27:21 UTC Duration: 514 seconds Status: Pass Rate: 0% (0/1) #passed tests/ total tests Compliance : NO #final result of this compliance Overview: dovetail.ipv6.tc001(2/3) # passed sub tests/ total sub tests -objective: VIM ipv6 operations, to create/delete network, port and subnet - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network pass - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port fail - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet pass Welcome to share your ideas within the mailing list or the wiki page https://wiki.opnfv.org/pages/editpage.action?pageId=8685074 BR Leo Wang _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
