Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
On 29 Jun 2021, at 20:49, Amber, Kumar wrote:
> Hi Flavio,
>
> Comments Inline.
>
>> -Original Message-
>> From: Flavio Leitner
>> Sent: Tuesday, June 29, 2021 11:49 PM
>> To: Amber, Kumar
>> Cc: Eelco Chaudron ; d...@openvswitch.org;
>> i.maxim...@ovn.org
>> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
>> autovalidator
>>
>> On Tue, Jun 29, 2021 at 05:11:00PM +, Amber, Kumar wrote:
>>> Hi Eelco, Flavio,
>>>
>>> Pls find my replies Inline
>>>
>>>> -Original Message-
>>>> From: Flavio Leitner
>>>> Sent: Tuesday, June 29, 2021 7:51 PM
>>>> To: Eelco Chaudron
>>>> Cc: Amber, Kumar ; Van Haaren, Harry
>>>> ; d...@openvswitch.org;
>>>> i.maxim...@ovn.org
>>>> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for
>>>> mfex autovalidator
>>>>
>>>> On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
>>>>>
>>>>>
>>>>> On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
>>>>>>> Tests:
>>>>>>> 6: OVS-DPDK - MFEX Autovalidator
>>>>>>> 7: OVS-DPDK - MFEX Autovalidator Fuzzy
>>>>>>>
>>>>>>> Added a new directory to store the PCAP file used in the tests
>>>>>>> and a script to generate the fuzzy traffic type pcap to be used
>>>>>>> in fuzzy unit test.
>>>>>>
>>>>>>
>>>>>> I haven't tried this yet but am I right that these tests are
>>>>>> going to pass a pcap to send traffic in a busy loop for 5
>>>>>> seconds in the first case and 20 seconds in the second case?
>>>>>>
>>>>>> I see that when autovalidator is set OVS will crash if one
>>>>>> implementation returns a different value, so I wonder why we
>>>>>> need to run for that long.
>>>>>
>>>>> I think we should remove the assert (already suggested by Harry),
>>>>> so it will not crass by accident if someone selects autovalidator
>>>>> in the field (and runs into an issue).
>>>>> Failure will then be detected by the ERROR log entries on shutdown.
>>>>
>>>> That's true for the testsuite, but not in production as there is
>>>> nothing to disable that.
>>>>
>>>> Perhaps if autovalidator detects an issue, it should log an ERROR
>>>> level log to report to testsuite, disable the failing mode and make
>>>> sure OVS is either in default or in another functional mode.
>>>
>>> So I have put the following :
>>> Removed the assert
>>> Allow the Auto-validator to run for all implementation and for a full
>> batch
>>> Document error via Vlog_Error
>>> Set the auto-validator to default {Scalar} when returning out in case
>> of failure.
>>
>> Sounds like a plan to me.
>> Is that okay with you Eelco?
ACK, this sounds good to me.
>>>>> I’m wondering if there is another way than a simple delay, as
>>>>> these tend to
>>>> cause issues later on. Can we check packets processed or something?
>>>>
>>>> Yeah, maybe we can pass all packets like 5x at least.
>>>
>>> Sure I will try to find something to do it more nicely.
>>> But just a thought keeping it 20sec allows for a full-stabilization and also
>> thorough testing of stability as well.
>>> So keeping it may not be just a bad idea.
>>
>> The issue is that if every test decides to delay seconds, the testsuite
>> becomes impractical. We have removed 'sleep' over time. Instead, we have
>> functions to wait for a certain cmdline output, or some event.
>> Yes, there are still some left to be fixed.
>>
>> Back to the point, maybe there is a signal of some sort we can get that
>> indicates the stability you're looking for.
>>
>
> I agree to the point and I am looking for a singal but currently due to
> assert removal we don’t have any marker.
> To Minimize the time, I did analysis of the time taken in each test-case :
>
> 1) for the simple test-case we don’t need the 5sec wait time as PCAP only
> contains one traffic or each type.
> 2) for fuzzy we do need at least 5sec for all 10k packets to be sent at-least
> 2x and also stability.
Guess you know the number of packets, so you could check in a 1sec delay loop
the number of received packets for the flows, and exit if you saw enough. And
just to be sure, you terminate after 5~10 seconds?
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
Hi Flavio,
Comments Inline.
> -Original Message-
> From: Flavio Leitner
> Sent: Tuesday, June 29, 2021 11:49 PM
> To: Amber, Kumar
> Cc: Eelco Chaudron ; d...@openvswitch.org;
> i.maxim...@ovn.org
> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
> autovalidator
>
> On Tue, Jun 29, 2021 at 05:11:00PM +, Amber, Kumar wrote:
> > Hi Eelco, Flavio,
> >
> > Pls find my replies Inline
> >
> > > -Original Message-
> > > From: Flavio Leitner
> > > Sent: Tuesday, June 29, 2021 7:51 PM
> > > To: Eelco Chaudron
> > > Cc: Amber, Kumar ; Van Haaren, Harry
> > > ; d...@openvswitch.org;
> > > i.maxim...@ovn.org
> > > Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for
> > > mfex autovalidator
> > >
> > > On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
> > > >
> > > >
> > > > On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> > > > >> Tests:
> > > > >> 6: OVS-DPDK - MFEX Autovalidator
> > > > >> 7: OVS-DPDK - MFEX Autovalidator Fuzzy
> > > > >>
> > > > >> Added a new directory to store the PCAP file used in the tests
> > > > >> and a script to generate the fuzzy traffic type pcap to be used
> > > > >> in fuzzy unit test.
> > > > >
> > > > >
> > > > > I haven't tried this yet but am I right that these tests are
> > > > > going to pass a pcap to send traffic in a busy loop for 5
> > > > > seconds in the first case and 20 seconds in the second case?
> > > > >
> > > > > I see that when autovalidator is set OVS will crash if one
> > > > > implementation returns a different value, so I wonder why we
> > > > > need to run for that long.
> > > >
> > > > I think we should remove the assert (already suggested by Harry),
> > > > so it will not crass by accident if someone selects autovalidator
> > > > in the field (and runs into an issue).
> > > > Failure will then be detected by the ERROR log entries on shutdown.
> > >
> > > That's true for the testsuite, but not in production as there is
> > > nothing to disable that.
> > >
> > > Perhaps if autovalidator detects an issue, it should log an ERROR
> > > level log to report to testsuite, disable the failing mode and make
> > > sure OVS is either in default or in another functional mode.
> >
> > So I have put the following :
> > Removed the assert
> > Allow the Auto-validator to run for all implementation and for a full
> batch
> > Document error via Vlog_Error
> > Set the auto-validator to default {Scalar} when returning out in case
> of failure.
>
> Sounds like a plan to me.
> Is that okay with you Eelco?
>
>
> > > > I’m wondering if there is another way than a simple delay, as
> > > > these tend to
> > > cause issues later on. Can we check packets processed or something?
> > >
> > > Yeah, maybe we can pass all packets like 5x at least.
> >
> > Sure I will try to find something to do it more nicely.
> > But just a thought keeping it 20sec allows for a full-stabilization and also
> thorough testing of stability as well.
> > So keeping it may not be just a bad idea.
>
> The issue is that if every test decides to delay seconds, the testsuite
> becomes impractical. We have removed 'sleep' over time. Instead, we have
> functions to wait for a certain cmdline output, or some event.
> Yes, there are still some left to be fixed.
>
> Back to the point, maybe there is a signal of some sort we can get that
> indicates the stability you're looking for.
>
I agree to the point and I am looking for a singal but currently due to assert
removal we don’t have any marker.
To Minimize the time, I did analysis of the time taken in each test-case :
1) for the simple test-case we don’t need the 5sec wait time as PCAP only
contains one traffic or each type.
2) for fuzzy we do need at least 5sec for all 10k packets to be sent at-least
2x and also stability.
Will the reductions suffice for now until I find a way to remove it completely
from Fuzzy ?
Regards
Amber
> --
> fbl
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
On Tue, Jun 29, 2021 at 05:11:00PM +, Amber, Kumar wrote:
> Hi Eelco, Flavio,
>
> Pls find my replies Inline
>
> > -Original Message-
> > From: Flavio Leitner
> > Sent: Tuesday, June 29, 2021 7:51 PM
> > To: Eelco Chaudron
> > Cc: Amber, Kumar ; Van Haaren, Harry
> > ; d...@openvswitch.org; i.maxim...@ovn.org
> > Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
> > autovalidator
> >
> > On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
> > >
> > >
> > > On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> > > >> Tests:
> > > >> 6: OVS-DPDK - MFEX Autovalidator
> > > >> 7: OVS-DPDK - MFEX Autovalidator Fuzzy
> > > >>
> > > >> Added a new directory to store the PCAP file used in the tests and
> > > >> a script to generate the fuzzy traffic type pcap to be used in
> > > >> fuzzy unit test.
> > > >
> > > >
> > > > I haven't tried this yet but am I right that these tests are going
> > > > to pass a pcap to send traffic in a busy loop for 5 seconds in the
> > > > first case and 20 seconds in the second case?
> > > >
> > > > I see that when autovalidator is set OVS will crash if one
> > > > implementation returns a different value, so I wonder why we need to
> > > > run for that long.
> > >
> > > I think we should remove the assert (already suggested by Harry), so
> > > it will not crass by accident if someone selects autovalidator in the
> > > field (and runs into an issue).
> > > Failure will then be detected by the ERROR log entries on shutdown.
> >
> > That's true for the testsuite, but not in production as there is nothing to
> > disable that.
> >
> > Perhaps if autovalidator detects an issue, it should log an ERROR level log
> > to
> > report to testsuite, disable the failing mode and make sure OVS is either in
> > default or in another functional mode.
>
> So I have put the following :
> Removed the assert
> Allow the Auto-validator to run for all implementation and for a full
> batch
> Document error via Vlog_Error
> Set the auto-validator to default {Scalar} when returning out in case
> of failure.
Sounds like a plan to me.
Is that okay with you Eelco?
> > > I’m wondering if there is another way than a simple delay, as these tend
> > > to
> > cause issues later on. Can we check packets processed or something?
> >
> > Yeah, maybe we can pass all packets like 5x at least.
>
> Sure I will try to find something to do it more nicely.
> But just a thought keeping it 20sec allows for a full-stabilization and also
> thorough testing of stability as well.
> So keeping it may not be just a bad idea.
The issue is that if every test decides to delay seconds, the testsuite
becomes impractical. We have removed 'sleep' over time. Instead, we
have functions to wait for a certain cmdline output, or some event.
Yes, there are still some left to be fixed.
Back to the point, maybe there is a signal of some sort we can get
that indicates the stability you're looking for.
--
fbl
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
Hi Eelco, Flavio,
Pls find my replies Inline
> -Original Message-
> From: Flavio Leitner
> Sent: Tuesday, June 29, 2021 7:51 PM
> To: Eelco Chaudron
> Cc: Amber, Kumar ; Van Haaren, Harry
> ; d...@openvswitch.org; i.maxim...@ovn.org
> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
> autovalidator
>
> On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
> >
> >
> > On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> >
> > > Hi,
> > >
> > >
> > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> > >> Tests:
> > >> 6: OVS-DPDK - MFEX Autovalidator
> > >> 7: OVS-DPDK - MFEX Autovalidator Fuzzy
> > >>
> > >> Added a new directory to store the PCAP file used in the tests and
> > >> a script to generate the fuzzy traffic type pcap to be used in
> > >> fuzzy unit test.
> > >
> > >
> > > I haven't tried this yet but am I right that these tests are going
> > > to pass a pcap to send traffic in a busy loop for 5 seconds in the
> > > first case and 20 seconds in the second case?
> > >
> > > I see that when autovalidator is set OVS will crash if one
> > > implementation returns a different value, so I wonder why we need to
> > > run for that long.
> >
> > I think we should remove the assert (already suggested by Harry), so
> > it will not crass by accident if someone selects autovalidator in the
> > field (and runs into an issue).
> > Failure will then be detected by the ERROR log entries on shutdown.
>
> That's true for the testsuite, but not in production as there is nothing to
> disable that.
>
> Perhaps if autovalidator detects an issue, it should log an ERROR level log to
> report to testsuite, disable the failing mode and make sure OVS is either in
> default or in another functional mode.
So I have put the following :
Removed the assert
Allow the Auto-validator to run for all implementation and for a full
batch
Document error via Vlog_Error
Set the auto-validator to default {Scalar} when returning out in case
of failure.
>
> > I’m wondering if there is another way than a simple delay, as these tend to
> cause issues later on. Can we check packets processed or something?
>
> Yeah, maybe we can pass all packets like 5x at least.
Sure I will try to find something to do it more nicely.
But just a thought keeping it 20sec allows for a full-stabilization and also
thorough testing of stability as well.
So keeping it may not be just a bad idea.
>
> fbl
>
>
> >
> > > It is storing a python tool in the pcap directory. I think the fuzzy
> > > tool could be called 'mfex_fuzzy.py' and stay in tests/ with other
> > > similar testing tools.
> > >
> > > Also, I don't think the test environment sets OVS_DIR. The 'tests/'
> > > is actually $srcdir, but I could be wrong here.
> > >
> > > BTW, scapy is not mandatory to build or test OVS, so if that tool is
> > > not available, the test should be skipped and not fail.
> > >
> > > Thanks,
> > > fbl
>
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
> Tests:
> 6: OVS-DPDK - MFEX Autovalidator
> 7: OVS-DPDK - MFEX Autovalidator Fuzzy
>
> Added a new directory to store the PCAP file used
> in the tests and a script to generate the fuzzy traffic
> type pcap to be used in fuzzy unit test.
Hi Amber,
I don't have much to add here that Flavio and Eelco have already flagged.
Looking forward to reviewing/testing the v5.
Regards
Ian
>
> Signed-off-by: Kumar Amber
> ---
> tests/automake.mk| 5 +
> tests/pcap/fuzzy.py | 32 ++
> tests/pcap/mfex_test | Bin 0 -> 416 bytes
> tests/system-dpdk.at | 46 +++
> 4 files changed, 83 insertions(+)
> create mode 100755 tests/pcap/fuzzy.py
> create mode 100644 tests/pcap/mfex_test
>
> diff --git a/tests/automake.mk b/tests/automake.mk
> index 1a528aa39..532875971 100644
> --- a/tests/automake.mk
> +++ b/tests/automake.mk
> @@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at:
> tests/automake.mk
> echo "TEST_FUZZ_REGRESSION([$$basename])"; \
> done > $@.tmp && mv $@.tmp $@
>
> +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
> +MFEX_AUTOVALIDATOR_TESTS = \
> + tests/pcap/mfex_test \
> + tests/pcap/fuzzy.py
> +
> OVSDB_CLUSTER_TESTSUITE_AT = \
> tests/ovsdb-cluster-testsuite.at \
> tests/ovsdb-execution.at \
> diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py
> new file mode 100755
> index 0..a8051ba2b
> --- /dev/null
> +++ b/tests/pcap/fuzzy.py
> @@ -0,0 +1,32 @@
> +#!/usr/bin/python3
> +try:
> + from scapy.all import *
> +except ModuleNotFoundError as err:
> + print(err + ": Scapy")
> +import sys
> +import os
> +
> +path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy"
> +pktdump = PcapWriter(path, append=False, sync=True)
> +
> +for i in range(0, 2000):
> +
> + # Generate random protocol bases, use a fuzz() over the combined packet
> for
> full fuzzing.
> + eth = Ether(src=RandMAC(), dst=RandMAC())
> + vlan = Dot1Q()
> + ipv4 = IP(src=RandIP(), dst=RandIP())
> + ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
> + udp = UDP()
> + tcp = TCP()
> +
> + # IPv4 packets with fuzzing
> + pktdump.write(fuzz(eth/ipv4/udp))
> + pktdump.write(fuzz(eth/ipv4/tcp))
> + pktdump.write(fuzz(eth/vlan/ipv4/udp))
> + pktdump.write(fuzz(eth/vlan/ipv4/tcp))
> +
> +# IPv6 packets with fuzzing
> + pktdump.write(fuzz(eth/ipv6/udp))
> + pktdump.write(fuzz(eth/ipv6/tcp))
> + pktdump.write(fuzz(eth/vlan/ipv6/udp))
> + pktdump.write(fuzz(eth/vlan/ipv6/tcp))
> \ No newline at end of file
> diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test
> new file mode 100644
> index
> ..1aac67b8d643ecb016c758cb
> a4cc32212a80f52a
> GIT binary patch
> literal 416
> zcmca|c+)~A1{MYw`2U}Qff2}QK`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}o
> |
> z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$ z
> xK>KST_NTIwYriok6N4Vm)gX-
> Q@c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
>
> literal 0
> HcmV?d1
>
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index 802895488..46eaea35a 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -232,3 +232,49 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open
> vSwitch kernel module is probably
> \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> AT_CLEANUP
> dnl
> --
> +
> +dnl
> --
> +dnl Add standard DPDK PHY port
> +AT_SETUP([OVS-DPDK - MFEX Autovalidator])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
> options:dpdk-
> devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/mfex_test,infinite_rx=1], [],
> [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +
> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0],
> [dnl
> +Miniflow implementation set to autovalidator.
> +])
> +sleep 5
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
> +AT_CLEANUP
> +dnl
> --
> +
> +dnl
> --
> +dnl Add standard DPDK PHY port
> +AT_SETUP([OVS-DPDK - MFEX Autovalidator Fuzzy])
> +AT_KEYWORDS([dpdk])
> +AT_CHECK([$PYTHON3 $OVS_DIR/tests/pcap/fuzzy.py], [], [stdout])
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
> options:dpdk-
> devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/fuzzy,infinite_rx=1], [],
> [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +
> +AT_CHECK([ovs-appctl dpif
Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote: > > > On 28 Jun 2021, at 4:57, Flavio Leitner wrote: > > > Hi, > > > > > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote: > >> Tests: > >> 6: OVS-DPDK - MFEX Autovalidator > >> 7: OVS-DPDK - MFEX Autovalidator Fuzzy > >> > >> Added a new directory to store the PCAP file used > >> in the tests and a script to generate the fuzzy traffic > >> type pcap to be used in fuzzy unit test. > > > > > > I haven't tried this yet but am I right that these tests are > > going to pass a pcap to send traffic in a busy loop for 5 > > seconds in the first case and 20 seconds in the second case? > > > > I see that when autovalidator is set OVS will crash if one > > implementation returns a different value, so I wonder why > > we need to run for that long. > > I think we should remove the assert (already suggested by Harry), > so it will not crass by accident if someone selects autovalidator > in the field (and runs into an issue). > Failure will then be detected by the ERROR log entries on shutdown. That's true for the testsuite, but not in production as there is nothing to disable that. Perhaps if autovalidator detects an issue, it should log an ERROR level log to report to testsuite, disable the failing mode and make sure OVS is either in default or in another functional mode. > I’m wondering if there is another way than a simple delay, as these tend to > cause issues later on. Can we check packets processed or something? Yeah, maybe we can pass all packets like 5x at least. fbl > > > It is storing a python tool in the pcap directory. I think the > > fuzzy tool could be called 'mfex_fuzzy.py' and stay in tests/ > > with other similar testing tools. > > > > Also, I don't think the test environment sets OVS_DIR. The > > 'tests/' is actually $srcdir, but I could be wrong here. > > > > BTW, scapy is not mandatory to build or test OVS, so if that > > tool is not available, the test should be skipped and not fail. > > > > Thanks, > > fbl ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> Hi,
>
>
> On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
>> Tests:
>> 6: OVS-DPDK - MFEX Autovalidator
>> 7: OVS-DPDK - MFEX Autovalidator Fuzzy
>>
>> Added a new directory to store the PCAP file used
>> in the tests and a script to generate the fuzzy traffic
>> type pcap to be used in fuzzy unit test.
>
>
> I haven't tried this yet but am I right that these tests are
> going to pass a pcap to send traffic in a busy loop for 5
> seconds in the first case and 20 seconds in the second case?
>
> I see that when autovalidator is set OVS will crash if one
> implementation returns a different value, so I wonder why
> we need to run for that long.
I think we should remove the assert (already suggested by Harry), so it will
not crass by accident if someone selects autovalidator in the field (and runs
into an issue).
Failure will then be detected by the ERROR log entries on shutdown.
I’m wondering if there is another way than a simple delay, as these tend to
cause issues later on. Can we check packets processed or something?
> It is storing a python tool in the pcap directory. I think the
> fuzzy tool could be called 'mfex_fuzzy.py' and stay in tests/
> with other similar testing tools.
>
> Also, I don't think the test environment sets OVS_DIR. The
> 'tests/' is actually $srcdir, but I could be wrong here.
>
> BTW, scapy is not mandatory to build or test OVS, so if that
> tool is not available, the test should be skipped and not fail.
>
> Thanks,
> fbl
>
>
>>
>> Signed-off-by: Kumar Amber
>> ---
>> tests/automake.mk| 5 +
>> tests/pcap/fuzzy.py | 32 ++
>> tests/pcap/mfex_test | Bin 0 -> 416 bytes
>> tests/system-dpdk.at | 46 +++
>> 4 files changed, 83 insertions(+)
>> create mode 100755 tests/pcap/fuzzy.py
>> create mode 100644 tests/pcap/mfex_test
>>
>> diff --git a/tests/automake.mk b/tests/automake.mk
>> index 1a528aa39..532875971 100644
>> --- a/tests/automake.mk
>> +++ b/tests/automake.mk
>> @@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at:
>> tests/automake.mk
>> echo "TEST_FUZZ_REGRESSION([$$basename])"; \
>> done > $@.tmp && mv $@.tmp $@
>>
>> +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
>> +MFEX_AUTOVALIDATOR_TESTS = \
>> +tests/pcap/mfex_test \
>> +tests/pcap/fuzzy.py
>> +
>> OVSDB_CLUSTER_TESTSUITE_AT = \
>> tests/ovsdb-cluster-testsuite.at \
>> tests/ovsdb-execution.at \
>> diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py
>> new file mode 100755
>> index 0..a8051ba2b
>> --- /dev/null
>> +++ b/tests/pcap/fuzzy.py
>> @@ -0,0 +1,32 @@
>> +#!/usr/bin/python3
>> +try:
>> + from scapy.all import *
>> +except ModuleNotFoundError as err:
>> + print(err + ": Scapy")
>> +import sys
>> +import os
>> +
>> +path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy"
>> +pktdump = PcapWriter(path, append=False, sync=True)
>> +
>> +for i in range(0, 2000):
>> +
>> + # Generate random protocol bases, use a fuzz() over the combined packet
>> for full fuzzing.
>> + eth = Ether(src=RandMAC(), dst=RandMAC())
>> + vlan = Dot1Q()
>> + ipv4 = IP(src=RandIP(), dst=RandIP())
>> + ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
>> + udp = UDP()
>> + tcp = TCP()
>> +
>> + # IPv4 packets with fuzzing
>> + pktdump.write(fuzz(eth/ipv4/udp))
>> + pktdump.write(fuzz(eth/ipv4/tcp))
>> + pktdump.write(fuzz(eth/vlan/ipv4/udp))
>> + pktdump.write(fuzz(eth/vlan/ipv4/tcp))
>> +
>> +# IPv6 packets with fuzzing
>> + pktdump.write(fuzz(eth/ipv6/udp))
>> + pktdump.write(fuzz(eth/ipv6/tcp))
>> + pktdump.write(fuzz(eth/vlan/ipv6/udp))
>> + pktdump.write(fuzz(eth/vlan/ipv6/tcp))
>> \ No newline at end of file
>> diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test
>> new file mode 100644
>> index
>> ..1aac67b8d643ecb016c758cba4cc32212a80f52a
>> GIT binary patch
>> literal 416
>> zcmca|c+)~A1{MYw`2U}Qff2}QK`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}o|
>> z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$> xK>KST_NTIwYriok6N4Vm)gX-Q@c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
>>
>> literal 0
>> HcmV?d1
>>
>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
>> index 802895488..46eaea35a 100644
>> --- a/tests/system-dpdk.at
>> +++ b/tests/system-dpdk.at
>> @@ -232,3 +232,49 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch
>> kernel module is probably
>> \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
>> AT_CLEANUP
>> dnl
>> --
>> +
>> +dnl
>> --
>> +dnl Add standard DPDK PHY port
>> +AT_SETUP([OVS-DPDK - MFEX Autovalidator])
>> +AT_KEYWORDS([dpdk])
>> +
>> +OVS_DPDK_START()
>> +
>> +dnl Add userspace bridge and attach it to OVS
>> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
>> +AT_
Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
Hi Flavio, Thanks again for excellent reviews and my replies are inline. > -Original Message- > From: Flavio Leitner > Sent: Monday, June 28, 2021 8:28 AM > To: Amber, Kumar > Cc: d...@openvswitch.org; i.maxim...@ovn.org > Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex > autovalidator > > Hi, > > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote: > > Tests: > > 6: OVS-DPDK - MFEX Autovalidator > > 7: OVS-DPDK - MFEX Autovalidator Fuzzy > > > > Added a new directory to store the PCAP file used in the tests and a > > script to generate the fuzzy traffic type pcap to be used in fuzzy > > unit test. > > > I haven't tried this yet but am I right that these tests are going to pass a > pcap > to send traffic in a busy loop for 5 seconds in the first case and 20 seconds > in > the second case? Yes , The second test is supposed to send traffic via a PCAP which holds over 10k Different traffic type and thus we ensure by a time delay that each one of them is send Before exiting the test. > > I see that when autovalidator is set OVS will crash if one implementation > returns a different value, so I wonder why we need to run for that long. > > It is storing a python tool in the pcap directory. I think the fuzzy tool > could > be called 'mfex_fuzzy.py' and stay in tests/ with other similar testing tools. > Make sense renamed and moved to test in v5. > Also, I don't think the test environment sets OVS_DIR. The 'tests/' is > actually > $srcdir, but I could be wrong here. > No correct changed to $srcdir and it makes test robust too thanks 😊 > BTW, scapy is not mandatory to build or test OVS, so if that tool is not > available, the test should be skipped and not fail. > Done and fixed in v5. > Thanks, > fbl > > > > > > Signed-off-by: Kumar Amber > > --- > > tests/automake.mk| 5 + > > tests/pcap/fuzzy.py | 32 ++ > > tests/pcap/mfex_test | Bin 0 -> 416 bytes tests/system-dpdk.at | 46 > > +++ > > 4 files changed, 83 insertions(+) > > create mode 100755 tests/pcap/fuzzy.py create mode 100644 > > tests/pcap/mfex_test > > > > diff --git a/tests/automake.mk b/tests/automake.mk index > > 1a528aa39..532875971 100644 > > --- a/tests/automake.mk > > +++ b/tests/automake.mk > > @@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at: > tests/automake.mk > > echo "TEST_FUZZ_REGRESSION([$$basename])"; \ > > done > $@.tmp && mv $@.tmp $@ > > > > +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS) > MFEX_AUTOVALIDATOR_TESTS = > > +\ > > + tests/pcap/mfex_test \ > > + tests/pcap/fuzzy.py > > + > > OVSDB_CLUSTER_TESTSUITE_AT = \ > > tests/ovsdb-cluster-testsuite.at \ > > tests/ovsdb-execution.at \ > > diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py new file mode > > 100755 index 0..a8051ba2b > > --- /dev/null > > +++ b/tests/pcap/fuzzy.py > > @@ -0,0 +1,32 @@ > > +#!/usr/bin/python3 > > +try: > > + from scapy.all import * > > +except ModuleNotFoundError as err: > > + print(err + ": Scapy") > > +import sys > > +import os > > + > > +path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy" > > +pktdump = PcapWriter(path, append=False, sync=True) > > + > > +for i in range(0, 2000): > > + > > + # Generate random protocol bases, use a fuzz() over the combined > packet for full fuzzing. > > + eth = Ether(src=RandMAC(), dst=RandMAC()) > > + vlan = Dot1Q() > > + ipv4 = IP(src=RandIP(), dst=RandIP()) > > + ipv6 = IPv6(src=RandIP6(), dst=RandIP6()) > > + udp = UDP() > > + tcp = TCP() > > + > > + # IPv4 packets with fuzzing > > + pktdump.write(fuzz(eth/ipv4/udp)) > > + pktdump.write(fuzz(eth/ipv4/tcp)) > > + pktdump.write(fuzz(eth/vlan/ipv4/udp)) > > + pktdump.write(fuzz(eth/vlan/ipv4/tcp)) > > + > > +# IPv6 packets with fuzzing > > + pktdump.write(fuzz(eth/ipv6/udp)) > > + pktdump.write(fuzz(eth/ipv6/tcp)) > > + pktdump.write(fuzz(eth/vlan/ipv6/udp)) > > + pktdump.write(fuzz(eth/vlan/ipv6/tcp)) > > \ No newline at end of file > > diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test new file mode > > 100644 index > > > ..1aac67b8d643ec
Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
Hi,
On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> Tests:
> 6: OVS-DPDK - MFEX Autovalidator
> 7: OVS-DPDK - MFEX Autovalidator Fuzzy
>
> Added a new directory to store the PCAP file used
> in the tests and a script to generate the fuzzy traffic
> type pcap to be used in fuzzy unit test.
I haven't tried this yet but am I right that these tests are
going to pass a pcap to send traffic in a busy loop for 5
seconds in the first case and 20 seconds in the second case?
I see that when autovalidator is set OVS will crash if one
implementation returns a different value, so I wonder why
we need to run for that long.
It is storing a python tool in the pcap directory. I think the
fuzzy tool could be called 'mfex_fuzzy.py' and stay in tests/
with other similar testing tools.
Also, I don't think the test environment sets OVS_DIR. The
'tests/' is actually $srcdir, but I could be wrong here.
BTW, scapy is not mandatory to build or test OVS, so if that
tool is not available, the test should be skipped and not fail.
Thanks,
fbl
>
> Signed-off-by: Kumar Amber
> ---
> tests/automake.mk| 5 +
> tests/pcap/fuzzy.py | 32 ++
> tests/pcap/mfex_test | Bin 0 -> 416 bytes
> tests/system-dpdk.at | 46 +++
> 4 files changed, 83 insertions(+)
> create mode 100755 tests/pcap/fuzzy.py
> create mode 100644 tests/pcap/mfex_test
>
> diff --git a/tests/automake.mk b/tests/automake.mk
> index 1a528aa39..532875971 100644
> --- a/tests/automake.mk
> +++ b/tests/automake.mk
> @@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at:
> tests/automake.mk
> echo "TEST_FUZZ_REGRESSION([$$basename])"; \
> done > $@.tmp && mv $@.tmp $@
>
> +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
> +MFEX_AUTOVALIDATOR_TESTS = \
> + tests/pcap/mfex_test \
> + tests/pcap/fuzzy.py
> +
> OVSDB_CLUSTER_TESTSUITE_AT = \
> tests/ovsdb-cluster-testsuite.at \
> tests/ovsdb-execution.at \
> diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py
> new file mode 100755
> index 0..a8051ba2b
> --- /dev/null
> +++ b/tests/pcap/fuzzy.py
> @@ -0,0 +1,32 @@
> +#!/usr/bin/python3
> +try:
> + from scapy.all import *
> +except ModuleNotFoundError as err:
> + print(err + ": Scapy")
> +import sys
> +import os
> +
> +path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy"
> +pktdump = PcapWriter(path, append=False, sync=True)
> +
> +for i in range(0, 2000):
> +
> + # Generate random protocol bases, use a fuzz() over the combined packet
> for full fuzzing.
> + eth = Ether(src=RandMAC(), dst=RandMAC())
> + vlan = Dot1Q()
> + ipv4 = IP(src=RandIP(), dst=RandIP())
> + ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
> + udp = UDP()
> + tcp = TCP()
> +
> + # IPv4 packets with fuzzing
> + pktdump.write(fuzz(eth/ipv4/udp))
> + pktdump.write(fuzz(eth/ipv4/tcp))
> + pktdump.write(fuzz(eth/vlan/ipv4/udp))
> + pktdump.write(fuzz(eth/vlan/ipv4/tcp))
> +
> +# IPv6 packets with fuzzing
> + pktdump.write(fuzz(eth/ipv6/udp))
> + pktdump.write(fuzz(eth/ipv6/tcp))
> + pktdump.write(fuzz(eth/vlan/ipv6/udp))
> + pktdump.write(fuzz(eth/vlan/ipv6/tcp))
> \ No newline at end of file
> diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test
> new file mode 100644
> index
> ..1aac67b8d643ecb016c758cba4cc32212a80f52a
> GIT binary patch
> literal 416
> zcmca|c+)~A1{MYw`2U}Qff2}QK`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}o|
> z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$ xK>KST_NTIwYriok6N4Vm)gX-Q@c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
>
> literal 0
> HcmV?d1
>
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index 802895488..46eaea35a 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -232,3 +232,49 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch
> kernel module is probably
> \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> AT_CLEANUP
> dnl
> --
> +
> +dnl
> --
> +dnl Add standard DPDK PHY port
> +AT_SETUP([OVS-DPDK - MFEX Autovalidator])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
> options:dpdk-devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/mfex_test,infinite_rx=1],
> [], [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +
> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0],
> [dnl
> +Miniflow implementation set to autovalidator.
> +])
> +sleep 5
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
> +AT_CLEANUP
> +dnl
> --
> +
> +dnl
> -
[ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator
Tests:
6: OVS-DPDK - MFEX Autovalidator
7: OVS-DPDK - MFEX Autovalidator Fuzzy
Added a new directory to store the PCAP file used
in the tests and a script to generate the fuzzy traffic
type pcap to be used in fuzzy unit test.
Signed-off-by: Kumar Amber
---
tests/automake.mk| 5 +
tests/pcap/fuzzy.py | 32 ++
tests/pcap/mfex_test | Bin 0 -> 416 bytes
tests/system-dpdk.at | 46 +++
4 files changed, 83 insertions(+)
create mode 100755 tests/pcap/fuzzy.py
create mode 100644 tests/pcap/mfex_test
diff --git a/tests/automake.mk b/tests/automake.mk
index 1a528aa39..532875971 100644
--- a/tests/automake.mk
+++ b/tests/automake.mk
@@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at: tests/automake.mk
echo "TEST_FUZZ_REGRESSION([$$basename])"; \
done > $@.tmp && mv $@.tmp $@
+EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
+MFEX_AUTOVALIDATOR_TESTS = \
+ tests/pcap/mfex_test \
+ tests/pcap/fuzzy.py
+
OVSDB_CLUSTER_TESTSUITE_AT = \
tests/ovsdb-cluster-testsuite.at \
tests/ovsdb-execution.at \
diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py
new file mode 100755
index 0..a8051ba2b
--- /dev/null
+++ b/tests/pcap/fuzzy.py
@@ -0,0 +1,32 @@
+#!/usr/bin/python3
+try:
+ from scapy.all import *
+except ModuleNotFoundError as err:
+ print(err + ": Scapy")
+import sys
+import os
+
+path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy"
+pktdump = PcapWriter(path, append=False, sync=True)
+
+for i in range(0, 2000):
+
+ # Generate random protocol bases, use a fuzz() over the combined packet for
full fuzzing.
+ eth = Ether(src=RandMAC(), dst=RandMAC())
+ vlan = Dot1Q()
+ ipv4 = IP(src=RandIP(), dst=RandIP())
+ ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
+ udp = UDP()
+ tcp = TCP()
+
+ # IPv4 packets with fuzzing
+ pktdump.write(fuzz(eth/ipv4/udp))
+ pktdump.write(fuzz(eth/ipv4/tcp))
+ pktdump.write(fuzz(eth/vlan/ipv4/udp))
+ pktdump.write(fuzz(eth/vlan/ipv4/tcp))
+
+# IPv6 packets with fuzzing
+ pktdump.write(fuzz(eth/ipv6/udp))
+ pktdump.write(fuzz(eth/ipv6/tcp))
+ pktdump.write(fuzz(eth/vlan/ipv6/udp))
+ pktdump.write(fuzz(eth/vlan/ipv6/tcp))
\ No newline at end of file
diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test
new file mode 100644
index
..1aac67b8d643ecb016c758cba4cc32212a80f52a
GIT binary patch
literal 416
zcmca|c+)~A1{MYw`2U}Qff2}QK`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}o|
z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$KST_NTIwYriok6N4Vm)gX-Q@c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
literal 0
HcmV?d1
diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
index 802895488..46eaea35a 100644
--- a/tests/system-dpdk.at
+++ b/tests/system-dpdk.at
@@ -232,3 +232,49 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch
kernel module is probably
\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
AT_CLEANUP
dnl --
+
+dnl --
+dnl Add standard DPDK PHY port
+AT_SETUP([OVS-DPDK - MFEX Autovalidator])
+AT_KEYWORDS([dpdk])
+
+OVS_DPDK_START()
+
+dnl Add userspace bridge and attach it to OVS
+AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
+AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
options:dpdk-devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/mfex_test,infinite_rx=1],
[], [stdout], [stderr])
+AT_CHECK([ovs-vsctl show], [], [stdout])
+
+
+AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], [dnl
+Miniflow implementation set to autovalidator.
+])
+sleep 5
+
+dnl Clean up
+AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
+AT_CLEANUP
+dnl --
+
+dnl --
+dnl Add standard DPDK PHY port
+AT_SETUP([OVS-DPDK - MFEX Autovalidator Fuzzy])
+AT_KEYWORDS([dpdk])
+AT_CHECK([$PYTHON3 $OVS_DIR/tests/pcap/fuzzy.py], [], [stdout])
+OVS_DPDK_START()
+
+dnl Add userspace bridge and attach it to OVS
+AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
+AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
options:dpdk-devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/fuzzy,infinite_rx=1],
[], [stdout], [stderr])
+AT_CHECK([ovs-vsctl show], [], [stdout])
+
+
+AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], [dnl
+Miniflow implementation set to autovalidator.
+])
+sleep 20
+
+dnl Clean up
+AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
+AT_CLEANUP
+dnl --
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
