Re: [ovs-dev] [v4 04/12] docs/dpdk/bridge: add miniflow extract section.
Hi Flavio , Thanks Again replies inline. > -Original Message- > From: Flavio Leitner > Sent: Monday, June 28, 2021 8:23 AM > To: Amber, Kumar > Cc: d...@openvswitch.org; i.maxim...@ovn.org > Subject: Re: [ovs-dev] [v4 04/12] docs/dpdk/bridge: add miniflow extract > section. > > On Thu, Jun 17, 2021 at 09:57:46PM +0530, Kumar Amber wrote: > > This commit adds a section to the dpdk/bridge.rst netdev > > documentation, detailing the added miniflow functionality. The newly > > added commands are documented, and sample output is provided. > > > > The use of auto-validator and special study function is also described > > in detail as well as running fuzzy tests. > > Usually we require to add the NEWS entry and documentation in the same > patch adding the feature because they help to document the patch and keep > sources integrity when bisecting. > Split NEWS and Documentation into patches in v5. > > > > Signed-off-by: Kumar Amber > > Co-authored-by: Cian Ferriter > > Signed-off-by: Cian Ferriter > > Co-authored-by: Harry van Haaren > > Signed-off-by: Harry van Haaren > > --- > > Documentation/topics/dpdk/bridge.rst | 105 > +++ > > NEWS | 3 + > > 2 files changed, 108 insertions(+) > > > > diff --git a/Documentation/topics/dpdk/bridge.rst > > b/Documentation/topics/dpdk/bridge.rst > > index f59e26cbe..b262b98f8 100644 > > --- a/Documentation/topics/dpdk/bridge.rst > > +++ b/Documentation/topics/dpdk/bridge.rst > > @@ -256,3 +256,108 @@ The following line should be seen in the > > configure output when the above option is used :: > > > > checking whether DPIF AVX512 is default implementation... yes > > + > > +Miniflow Extract > > + > > + > > +Miniflow extract (MFEX) performs parsing of the raw packets and > > +extracts the important header information into a compressed miniflow. > > +This miniflow is composed of bits and blocks where the bits signify > > +which blocks are set or have values where as the blocks hold the > > +metadata, ip, udp, vlan, etc. These values are used by the datapath for > switching decisions later. > > + > > +Most modern CPUs are have SIMD capabilities. These SIMD instructions > > +are able > > Have? > Fixed the typo. > > +to process a vector rather than act on one single data. OVS provides > > +multiple implementations of miniflow extract. This allows the user to > > +take advantage of SIMD instructions like AVX512 to gain additional > performance. > > + > > +A list of implementations can be obtained by the following command. > > +The command also shows whether the CPU supports each > implementation :: > > + > > +$ ovs-appctl dpif-netdev/miniflow-parser-get > > +Available Optimized Miniflow Extracts: > > + autovalidator (available: True) > > + disable (available: True) > > + study (available: True) > > + avx512_ip_udp (available: True) > > + > > +An implementation can be selected manually by the following command > :: > > + > > +$ ovs-appctl dpif-netdev/miniflow-parser-set study > > + > > +Also user can select the study implementation which studies the > > +traffic for a specific number of packets by applying all availbale > > +implementaions of > > Typos already mentioned. > Fixed and moved to unit test commit. > > +miniflow extract and than chooses the one with most optimal result > > +for that traffic pattern. > > + > > +Miniflow Extract Validation > > +~~~ > > + > > +As multiple versions of miniflow extract can co-exist, each with > > +different CPU ISA optimizations, it is important to validate that > > +they all give the exact same results. To easily test all miniflow > > +implementations, an ``autovalidator`` implementation of the miniflow > > +exists. This implementation runs all other available miniflow extract > > +implementations, and verifies that the results are identical. > > + > > +Running the OVS unit tests with the autovalidator enabled ensures all > > +implementations provide the same results. > > + > > +To set the Miniflow autovalidator, use this command :: > > + > > +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator > > + > > +Unit Test Miniflow Extract > > +++ > > + > > +Unit test can also be used to test the workflow mentioned above by > > +ru
Re: [ovs-dev] [v4 04/12] docs/dpdk/bridge: add miniflow extract section.
On Thu, Jun 17, 2021 at 09:57:46PM +0530, Kumar Amber wrote: > This commit adds a section to the dpdk/bridge.rst netdev documentation, > detailing the added miniflow functionality. The newly added commands are > documented, and sample output is provided. > > The use of auto-validator and special study function is also described > in detail as well as running fuzzy tests. Usually we require to add the NEWS entry and documentation in the same patch adding the feature because they help to document the patch and keep sources integrity when bisecting. > > Signed-off-by: Kumar Amber > Co-authored-by: Cian Ferriter > Signed-off-by: Cian Ferriter > Co-authored-by: Harry van Haaren > Signed-off-by: Harry van Haaren > --- > Documentation/topics/dpdk/bridge.rst | 105 +++ > NEWS | 3 + > 2 files changed, 108 insertions(+) > > diff --git a/Documentation/topics/dpdk/bridge.rst > b/Documentation/topics/dpdk/bridge.rst > index f59e26cbe..b262b98f8 100644 > --- a/Documentation/topics/dpdk/bridge.rst > +++ b/Documentation/topics/dpdk/bridge.rst > @@ -256,3 +256,108 @@ The following line should be seen in the configure > output when the above option > is used :: > > checking whether DPIF AVX512 is default implementation... yes > + > +Miniflow Extract > + > + > +Miniflow extract (MFEX) performs parsing of the raw packets and extracts the > +important header information into a compressed miniflow. This miniflow is > +composed of bits and blocks where the bits signify which blocks are set or > +have values where as the blocks hold the metadata, ip, udp, vlan, etc. These > +values are used by the datapath for switching decisions later. > + > +Most modern CPUs are have SIMD capabilities. These SIMD instructions are able Have? > +to process a vector rather than act on one single data. OVS provides multiple > +implementations of miniflow extract. This allows the user to take advantage > +of SIMD instructions like AVX512 to gain additional performance. > + > +A list of implementations can be obtained by the following command. The > +command also shows whether the CPU supports each implementation :: > + > +$ ovs-appctl dpif-netdev/miniflow-parser-get > +Available Optimized Miniflow Extracts: > + autovalidator (available: True) > + disable (available: True) > + study (available: True) > + avx512_ip_udp (available: True) > + > +An implementation can be selected manually by the following command :: > + > +$ ovs-appctl dpif-netdev/miniflow-parser-set study > + > +Also user can select the study implementation which studies the traffic for > +a specific number of packets by applying all availbale implementaions of Typos already mentioned. > +miniflow extract and than chooses the one with most optimal result for that > +traffic pattern. > + > +Miniflow Extract Validation > +~~~ > + > +As multiple versions of miniflow extract can co-exist, each with different > +CPU ISA optimizations, it is important to validate that they all give the > +exact same results. To easily test all miniflow implementations, an > +``autovalidator`` implementation of the miniflow exists. This implementation > +runs all other available miniflow extract implementations, and verifies that > +the results are identical. > + > +Running the OVS unit tests with the autovalidator enabled ensures all > +implementations provide the same results. > + > +To set the Miniflow autovalidator, use this command :: > + > +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator > + > +Unit Test Miniflow Extract > +++ > + > +Unit test can also be used to test the workflow mentioned above by running > +the following test-case in tests/system-dpdk.at :: > + > +make check-dpdk TESTSUITEFLAGS=6 > +6: OVS-DPDK - MFEX Autovalidator This will change over time. Can we use -k instead? > + > +The unit test uses mulitple traffic type to test the correctness of the > +implementaions. > + > +Running Fuzzy test with Autovalidator > ++ > + > +Fuzzy tests can also be done on minfilow extract with the help of Typo. > +auto-validator and Scapy. The steps below describes the steps to > +reproduce the setup with IP being fuzzed to generate packets. > + > +Scapy is used to create fuzzy IP packets and save them into a PCAP :: > + > +pkt = fuzz(Ether()/IP()/TCP()) > + > +Set the miniflow extract to autovalidator using :: > + > +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator > + > +OVS is configured to receive the generated packets :: > + > +$ ovs-vsctl add-port br0 pcap0 -- \ > +set Interface pcap0 type=dpdk options:dpdk-devargs=net_pcap0 > +"rx_pcap=fuzzy.pcap" This comes back to the point of adding the doc along with the test because I can't see how the test works in this patch. > + > +W
Re: [ovs-dev] [v4 04/12] docs/dpdk/bridge: add miniflow extract section.
Hi Ian, Thanks and inline are replies. > Also is there a patch to update the OVS manpages for these commands also? > Possibly later in the series? Just a small update later regarding packet study counter nothing more . > > > Signed-off-by: Kumar Amber > > Co-authored-by: Cian Ferriter > > Signed-off-by: Cian Ferriter > > Co-authored-by: Harry van Haaren > > Signed-off-by: Harry van Haaren > > --- > > Documentation/topics/dpdk/bridge.rst | 105 > +++ > > NEWS | 3 + > > 2 files changed, 108 insertions(+) > > > > diff --git a/Documentation/topics/dpdk/bridge.rst > > b/Documentation/topics/dpdk/bridge.rst > > index f59e26cbe..b262b98f8 100644 > > --- a/Documentation/topics/dpdk/bridge.rst > > +++ b/Documentation/topics/dpdk/bridge.rst > > @@ -256,3 +256,108 @@ The following line should be seen in the > > configure output when the above option is used :: > > > > checking whether DPIF AVX512 is default implementation... yes > > + > > +Miniflow Extract > > + > > + > > +Miniflow extract (MFEX) performs parsing of the raw packets and > > +extracts the important header information into a compressed miniflow. > > +This miniflow is composed of bits and blocks where the bits signify > > +which blocks are set or have values where as the blocks hold the > > +metadata, ip, udp, vlan, etc. These values are used by the datapath for > switching decisions later. > > + > > +Most modern CPUs are have SIMD capabilities. These SIMD instructions > > +are > > able > > +to process a vector rather than act on one single data. OVS provides > > +multiple implementations of miniflow extract. This allows the user to > > +take advantage of SIMD instructions like AVX512 to gain additional > performance. > > + > > +A list of implementations can be obtained by the following command. > > +The command also shows whether the CPU supports each > implementation :: > > + > > +$ ovs-appctl dpif-netdev/miniflow-parser-get > > +Available Optimized Miniflow Extracts: > > + autovalidator (available: True) > > + disable (available: True) > > + study (available: True) > > + avx512_ip_udp (available: True) > > + > > +An implementation can be selected manually by the following command > :: > > + > > +$ ovs-appctl dpif-netdev/miniflow-parser-set study > > + > > +Also user can select the study implementation which studies the > > +traffic for a specific number of packets by applying all availbale > > +implementaions of > > Typo above "available implementations" Fixed in v5. > > +miniflow extract and than chooses the one with most optimal result > > +for that > Would change than->then above > > +traffic pattern. > > + > > +Miniflow Extract Validation > > +~~~ > > + > > +As multiple versions of miniflow extract can co-exist, each with > > +different CPU ISA optimizations, it is important to validate that > > +they all give the exact same results. To easily test all miniflow > > +implementations, an ``autovalidator`` implementation of the miniflow > > +exists. This implementation runs all other available miniflow extract > > +implementations, and verifies that the results are identical. > > + > > +Running the OVS unit tests with the autovalidator enabled ensures all > > +implementations provide the same results. > > + > > +To set the Miniflow autovalidator, use this command :: > > + > > +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator > > + > > +Unit Test Miniflow Extract > > +++ > > + > > +Unit test can also be used to test the workflow mentioned above by > > +running the following test-case in tests/system-dpdk.at :: > > + > > +make check-dpdk TESTSUITEFLAGS=6 > > +6: OVS-DPDK - MFEX Autovalidator > > + > > +The unit test uses mulitple traffic type to test the correctness of > > +the implementaions. > > + > > +Running Fuzzy test with Autovalidator > > ++ > > + > > +Fuzzy tests can also be done on minfilow extract with the help of > > +auto-validator and Scapy. The steps below describes the steps to > > +reproduce the setup with IP being fuzzed to generate packets. > > + > > +Scapy is used to create fuzzy IP packets and save them into a PCAP :: > > + > > +pkt = fuzz(Ether()/IP()/TCP()) > > + > > +Set the miniflow extract to autovalidator using :: > > + > > +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator > > + > > +OVS is configured to receive the generated packets :: > > + > > +$ ovs-vsctl add-port br0 pcap0 -- \ > > +set Interface pcap0 type=dpdk options:dpdk-devargs=net_pcap0 > > +"rx_pcap=fuzzy.pcap" > > + > > +With this workflow, the autovalidator will ensure that all MFEX > > +implementations are classifying each packet in exactly the same way. > > +If an optimized MFEX implementation causes a different miniflow to be > > +generated, the autovalidator has ovs_assert an
Re: [ovs-dev] [v4 04/12] docs/dpdk/bridge: add miniflow extract section.
> This commit adds a section to the dpdk/bridge.rst netdev documentation, > detailing the added miniflow functionality. The newly added commands are > documented, and sample output is provided. > > The use of auto-validator and special study function is also described > in detail as well as running fuzzy tests. > Thanks for the patch Amber/Harry/Cian. Comments inline below. Also is there a patch to update the OVS manpages for these commands also? Possibly later in the series? > Signed-off-by: Kumar Amber > Co-authored-by: Cian Ferriter > Signed-off-by: Cian Ferriter > Co-authored-by: Harry van Haaren > Signed-off-by: Harry van Haaren > --- > Documentation/topics/dpdk/bridge.rst | 105 +++ > NEWS | 3 + > 2 files changed, 108 insertions(+) > > diff --git a/Documentation/topics/dpdk/bridge.rst > b/Documentation/topics/dpdk/bridge.rst > index f59e26cbe..b262b98f8 100644 > --- a/Documentation/topics/dpdk/bridge.rst > +++ b/Documentation/topics/dpdk/bridge.rst > @@ -256,3 +256,108 @@ The following line should be seen in the configure > output when the above option > is used :: > > checking whether DPIF AVX512 is default implementation... yes > + > +Miniflow Extract > + > + > +Miniflow extract (MFEX) performs parsing of the raw packets and extracts the > +important header information into a compressed miniflow. This miniflow is > +composed of bits and blocks where the bits signify which blocks are set or > +have values where as the blocks hold the metadata, ip, udp, vlan, etc. These > +values are used by the datapath for switching decisions later. > + > +Most modern CPUs are have SIMD capabilities. These SIMD instructions are > able > +to process a vector rather than act on one single data. OVS provides multiple > +implementations of miniflow extract. This allows the user to take advantage > +of SIMD instructions like AVX512 to gain additional performance. > + > +A list of implementations can be obtained by the following command. The > +command also shows whether the CPU supports each implementation :: > + > +$ ovs-appctl dpif-netdev/miniflow-parser-get > +Available Optimized Miniflow Extracts: > + autovalidator (available: True) > + disable (available: True) > + study (available: True) > + avx512_ip_udp (available: True) > + > +An implementation can be selected manually by the following command :: > + > +$ ovs-appctl dpif-netdev/miniflow-parser-set study > + > +Also user can select the study implementation which studies the traffic for > +a specific number of packets by applying all availbale implementaions of Typo above "available implementations" > +miniflow extract and than chooses the one with most optimal result for that Would change than->then above > +traffic pattern. > + > +Miniflow Extract Validation > +~~~ > + > +As multiple versions of miniflow extract can co-exist, each with different > +CPU ISA optimizations, it is important to validate that they all give the > +exact same results. To easily test all miniflow implementations, an > +``autovalidator`` implementation of the miniflow exists. This implementation > +runs all other available miniflow extract implementations, and verifies that > +the results are identical. > + > +Running the OVS unit tests with the autovalidator enabled ensures all > +implementations provide the same results. > + > +To set the Miniflow autovalidator, use this command :: > + > +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator > + > +Unit Test Miniflow Extract > +++ > + > +Unit test can also be used to test the workflow mentioned above by running > +the following test-case in tests/system-dpdk.at :: > + > +make check-dpdk TESTSUITEFLAGS=6 > +6: OVS-DPDK - MFEX Autovalidator > + > +The unit test uses mulitple traffic type to test the correctness of the > +implementaions. > + > +Running Fuzzy test with Autovalidator > ++ > + > +Fuzzy tests can also be done on minfilow extract with the help of > +auto-validator and Scapy. The steps below describes the steps to > +reproduce the setup with IP being fuzzed to generate packets. > + > +Scapy is used to create fuzzy IP packets and save them into a PCAP :: > + > +pkt = fuzz(Ether()/IP()/TCP()) > + > +Set the miniflow extract to autovalidator using :: > + > +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator > + > +OVS is configured to receive the generated packets :: > + > +$ ovs-vsctl add-port br0 pcap0 -- \ > +set Interface pcap0 type=dpdk options:dpdk-devargs=net_pcap0 > +"rx_pcap=fuzzy.pcap" > + > +With this workflow, the autovalidator will ensure that all MFEX > +implementations are classifying each packet in exactly the same way. > +If an optimized MFEX implementation causes a different miniflow to be > +generated, the autovalidator has ovs_ass
[ovs-dev] [v4 04/12] docs/dpdk/bridge: add miniflow extract section.
This commit adds a section to the dpdk/bridge.rst netdev documentation, detailing the added miniflow functionality. The newly added commands are documented, and sample output is provided. The use of auto-validator and special study function is also described in detail as well as running fuzzy tests. Signed-off-by: Kumar Amber Co-authored-by: Cian Ferriter Signed-off-by: Cian Ferriter Co-authored-by: Harry van Haaren Signed-off-by: Harry van Haaren --- Documentation/topics/dpdk/bridge.rst | 105 +++ NEWS | 3 + 2 files changed, 108 insertions(+) diff --git a/Documentation/topics/dpdk/bridge.rst b/Documentation/topics/dpdk/bridge.rst index f59e26cbe..b262b98f8 100644 --- a/Documentation/topics/dpdk/bridge.rst +++ b/Documentation/topics/dpdk/bridge.rst @@ -256,3 +256,108 @@ The following line should be seen in the configure output when the above option is used :: checking whether DPIF AVX512 is default implementation... yes + +Miniflow Extract + + +Miniflow extract (MFEX) performs parsing of the raw packets and extracts the +important header information into a compressed miniflow. This miniflow is +composed of bits and blocks where the bits signify which blocks are set or +have values where as the blocks hold the metadata, ip, udp, vlan, etc. These +values are used by the datapath for switching decisions later. + +Most modern CPUs are have SIMD capabilities. These SIMD instructions are able +to process a vector rather than act on one single data. OVS provides multiple +implementations of miniflow extract. This allows the user to take advantage +of SIMD instructions like AVX512 to gain additional performance. + +A list of implementations can be obtained by the following command. The +command also shows whether the CPU supports each implementation :: + +$ ovs-appctl dpif-netdev/miniflow-parser-get +Available Optimized Miniflow Extracts: + autovalidator (available: True) + disable (available: True) + study (available: True) + avx512_ip_udp (available: True) + +An implementation can be selected manually by the following command :: + +$ ovs-appctl dpif-netdev/miniflow-parser-set study + +Also user can select the study implementation which studies the traffic for +a specific number of packets by applying all availbale implementaions of +miniflow extract and than chooses the one with most optimal result for that +traffic pattern. + +Miniflow Extract Validation +~~~ + +As multiple versions of miniflow extract can co-exist, each with different +CPU ISA optimizations, it is important to validate that they all give the +exact same results. To easily test all miniflow implementations, an +``autovalidator`` implementation of the miniflow exists. This implementation +runs all other available miniflow extract implementations, and verifies that +the results are identical. + +Running the OVS unit tests with the autovalidator enabled ensures all +implementations provide the same results. + +To set the Miniflow autovalidator, use this command :: + +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator + +Unit Test Miniflow Extract +++ + +Unit test can also be used to test the workflow mentioned above by running +the following test-case in tests/system-dpdk.at :: + +make check-dpdk TESTSUITEFLAGS=6 +6: OVS-DPDK - MFEX Autovalidator + +The unit test uses mulitple traffic type to test the correctness of the +implementaions. + +Running Fuzzy test with Autovalidator ++ + +Fuzzy tests can also be done on minfilow extract with the help of +auto-validator and Scapy. The steps below describes the steps to +reproduce the setup with IP being fuzzed to generate packets. + +Scapy is used to create fuzzy IP packets and save them into a PCAP :: + +pkt = fuzz(Ether()/IP()/TCP()) + +Set the miniflow extract to autovalidator using :: + +$ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator + +OVS is configured to receive the generated packets :: + +$ ovs-vsctl add-port br0 pcap0 -- \ +set Interface pcap0 type=dpdk options:dpdk-devargs=net_pcap0 +"rx_pcap=fuzzy.pcap" + +With this workflow, the autovalidator will ensure that all MFEX +implementations are classifying each packet in exactly the same way. +If an optimized MFEX implementation causes a different miniflow to be +generated, the autovalidator has ovs_assert and logging statements that +will inform about the issue. + +Unit Fuzzy test with Autovalidator ++ + +The prerquiste before running the unit test is to run the script provided :: + +tests/pcap/fuzzy.py + +This script generates a pcap with mulitple type of fuzzed packets to be used +in the below unit test-case. + +Unit test can also be used to test the workflow mentioned above by running +the following test-case in tests/sy
