Adding vpp-dev. Following reviews in csit-dev and CSIT project calls, here is an updated proposal for a new set of FD.io<http://FD.io> VPP integration tests. The idea is to, over time, migrate the VIRL test cases to VPP_Path and VPP_Device integration tests describe below.
Test coverage gaps between existing VIRL and VPP make_test tests have been documented in an online sheet [1]. Many thanks to Jan Gelety for the analysis and CSIT contributors for their reviews. Unless issues/objections, plan is to progress this work during rls1807 dev cycle. Hope this makes sense, let us know if questions. -Maciek (CSIT PTL) [1] vpp-csit-verify-virl-test-gaps, https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing <snip> VPP_Path Integration Tests -------------------------- * Objective: (Integration) Functional acceptance of VPP feature paths driven by use cases. . Use cases defined as combinations of configurations and offered loads invoking multiple VPP packet processing paths. . Tester: "Does it do what is expected based on configuration and offered load?" . Includes regression tests based on field found bugs. * Environment: VPP make_test. . VPP test framework based on Python unittest. * Testing: VPP SW only. . api: vpp-papi. . dp: pg-rx-{vpp_graph_nodes_path}–tx-pg. * Results: Test [PASS|FAIL]. * Analytics: gcc gcov for code line coverage. * Continuous: FD.io<http://FD.io> per vpp patch execution with jjb verify voting. . Separate job from current vpp make verify(?). * Availability: New development in FD.io<http://FD.io> CSIT. . Partial coverage in VPP_Multi_Node Tests. * Project: FD.io<http://FD.io> CSIT. . Plan: introduce in rls1807, test coverage driven by priority use cases and VIRL equivalence. . Future: .. Negative tests: API fuzzing (/Fuzzapi/API-fuzzer), packet/protocol fuzzing (Cisco-Talos/mutiny-fuzzer). VPP_Device Integration Tests ---------------------------- * Objective: (Integration) Functional acceptance of VPP device driver interactions. . Tester: "Known VPP device and feature interactions must work!" * Environment: FD.io<http://FD.io> CSIT-CPL labs. . CSIT Python and Robot Framework libraries and tests. * Testing: VPP SW on compute HW with network devices. . VPP SW: .. api: vpp-papi. .. dp: int-dev–rx-{simple_VPP_graph_nodes_path}-tx-dev-int. . Compute HW: .. Xeon-Skylake, Arm and Atom nodes. .. Cisco, Intel, Mellanox NICs. * Results: Test [PASS|FAIL]. * Analytics: None. * Continuous: FD.io<http://FD.io> per vpp patch execution with jjb verify voting. * Availability: Depends on server and processor types. . Xeon: .. rls1807 - initial VPP and Scapy Pod/Container POC with VF drivers. .. Future - per vpp patch execution with jjb verify voting. . Arm: .. rls1807 - initial VPP and Scapy Pod/Container POC with VF drivers. .. Future - per vpp patch execution with jjb verify voting. . Atom: TBC. * Project: FD.io<http://FD.io> CSIT. . Plan: progress in rls1807. </snip> On 6 Jun 2018, at 14:25, Maciek Konstantynowicz (mkonstan) via Lists.Fd.Io <mkonstan=cisco....@lists.fd.io<mailto:mkonstan=cisco....@lists.fd.io>> wrote: Thanks for all comments! Here is an updated summary of both new test categories based on received feedback. (I removed the dependencies for clarity). I’m also sending a separate email with a straw-man proposal for parallelising functional HW device tests in VPP_Device. <snip> VPP_Path Integration Tests -------------------------- * Objective: (Integration) Functional acceptance of VPP feature paths driven by use cases (combinations of multiple nodes, configurations and offered loads). . Tester: "Must not break known node and feature interactions!" . Includes regression tests based on field reported bugs. * Environment: VPP make_test. * Testing: VPP SW only. . api: vpp-papi. . dp: pg-rx-{vpp_graph_nodes_path}–tx-pg. * Results: Test [PASS|FAIL]. * Analytics: gcc gcov for code line coverage. * Continuous: FD.io<http://fd.io/> per vpp patch execution with jjb verify voting. . Separate job from current vpp make verify. * Availability: New development in FD.io<http://fd.io/> CSIT. . Partial coverage in VPP_Multi_Node Tests. * Project: FD.io<http://fd.io/> CSIT. . Plan: introduce in rls1807, drive coverage driven by priority use cases and VIRL equivalence. VPP_Device Integration Tests ---------------------------- * Objective: (Integration) Functional acceptance of VPP device driver interactions. . Tester: "Must not break known device and feature interactions!" * Environment: FD.io<http://fd.io/> CSIT-CPL labs. * Testing: VPP SW on compute HW with network devices. . VPP SW: .. api: vpp-papi. .. dp: int-dev–rx-{simple_VPP_graph_nodes_path}-tx-dev-int. . Compute HW: .. Xeon-Skylake, Arm and Atom nodes. .. Cisco, Intel, Mellanox NICs. * Results: Test [PASS|FAIL]. * Analytics: None. * Continuous: FD.io<http://fd.io/> per vpp patch execution with jjb verify voting. * Availability: Depends on server and processor types. . Xeon: .. rls1807 - initial VPP and Scapy Pod/Container POC with VF drivers. .. Future - per vpp patch execution with jjb verify voting. . Arm: .. rls1807 - initial VPP and Scapy Pod/Container POC with VF drivers. .. Future - per vpp patch execution with jjb verify voting. . Atom: TBC. * Project: FD.io<http://fd.io/> CSIT. . Plan: progress in rls1807. </snip> -Maciek -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9590): https://lists.fd.io/g/vpp-dev/message/9590 Mute This Topic: https://lists.fd.io/mt/22001028/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
