Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
On 09/11/2018 04:15 PM, Ben Pfaff wrote: > Most of the test failures seem related to the "asynchronous message > control" tests. I haven't yet determined the reason for the failure, > but after some work I was able to reproduce it on my own system. Great, let me know when you've found out. I've uploaded 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-3 which should now work properly. I'll add dpdk support to it soon. Cheers, Thomas Goirand (zigo) ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
On Wed, Sep 05, 2018 at 01:55:08PM -0700, Ben Pfaff wrote: > On Wed, Sep 05, 2018 at 01:50:06PM +0200, Thomas Goirand wrote: > > On 09/04/2018 11:06 PM, Ben Pfaff wrote: > > > On Tue, Sep 04, 2018 at 09:20:45AM +0200, Thomas Goirand wrote: > > >> On 09/02/2018 03:12 AM, Justin Pettit wrote: > > >>> > > On Sep 1, 2018, at 3:52 PM, Ben Pfaff wrote: > > > > On Sat, Sep 01, 2018 at 01:23:32PM -0700, Justin Pettit wrote: > > > > > >> On Sep 1, 2018, at 12:21 PM, Thomas Goirand wrote: > > >> > > >> > > >> The only one failure: > > >> > > >> 2633: ovn -- ACL rate-limited logging FAILED > > >> (ovn.at:6516) > > > > > > My guess if that this is meter-related. Can you send the > > > ovs-vswitchd.log and testsuite.log so I can take a look? > > > > It probably hasn't changed from what he sent the first time around. > > >>> > > >>> Yes, "testsuite.log" was in the original message, so I don't need that. > > >>> Thomas, can you send me "ovs-vswitchd.log" and "ovn-controller.log"? > > >>> Does it consistently fail for you? > > >>> > > >>> --Justin > > >> > > >> Hi, > > >> > > >> As I blacklisted the above test, I uploaded to Sid, and now there's a > > >> number of failures on non-intel arch: > > >> > > >> https://buildd.debian.org/status/package.php?p=openvswitch > > >> https://buildd.debian.org/status/logs.php?pkg=openvswitch > > >> > > >> Ben, Justin, can you help me fix all of this? > > > > > > Thanks for passing that along. > > > > > > A lot of these failures seem to involve unexpected timeouts. I wonder > > > whether the buildds are so overloaded that some of the 10-second > > > timeouts in the testsuite are just too short. Usually, this is a > > > generous timeout interval. > > > > > > I sent a patch that should help to debug the problem by doing more > > > logging: > > > https://patchwork.ozlabs.org/patch/966087/ > > > > > > It won't help with tests that fully succeed, because the logs by default > > > are discarded, but for tests that have a sequence of waits, in which one > > > eventually fails, it will allow us to see how long the successful waits > > > took. > > > > > > Any chance you could apply that patch and try another build? Feel free > > > to wait for review, if you prefer. > > > > > > > Hi, > > > > I've just uploaded OVS with that patch. Thanks, I think it's a very good > > idea. And indeed, it looks like failing arch are the slower ones. > > I'm pretty pleased with the theory myself, but the results tend to show > that it wasn't the problem. In most of the tests that eventually > failed, the wait failure was preceded by other waits that succeeded > immediately, and the longest wait I see is 3 seconds. I'll look for > other possible causes. Most of the test failures seem related to the "asynchronous message control" tests. I haven't yet determined the reason for the failure, but after some work I was able to reproduce it on my own system. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
On Wed, Sep 05, 2018 at 01:50:06PM +0200, Thomas Goirand wrote: > On 09/04/2018 11:06 PM, Ben Pfaff wrote: > > On Tue, Sep 04, 2018 at 09:20:45AM +0200, Thomas Goirand wrote: > >> On 09/02/2018 03:12 AM, Justin Pettit wrote: > >>> > On Sep 1, 2018, at 3:52 PM, Ben Pfaff wrote: > > On Sat, Sep 01, 2018 at 01:23:32PM -0700, Justin Pettit wrote: > > > >> On Sep 1, 2018, at 12:21 PM, Thomas Goirand wrote: > >> > >> > >> The only one failure: > >> > >> 2633: ovn -- ACL rate-limited logging FAILED > >> (ovn.at:6516) > > > > My guess if that this is meter-related. Can you send the > > ovs-vswitchd.log and testsuite.log so I can take a look? > > It probably hasn't changed from what he sent the first time around. > >>> > >>> Yes, "testsuite.log" was in the original message, so I don't need that. > >>> Thomas, can you send me "ovs-vswitchd.log" and "ovn-controller.log"? > >>> Does it consistently fail for you? > >>> > >>> --Justin > >> > >> Hi, > >> > >> As I blacklisted the above test, I uploaded to Sid, and now there's a > >> number of failures on non-intel arch: > >> > >> https://buildd.debian.org/status/package.php?p=openvswitch > >> https://buildd.debian.org/status/logs.php?pkg=openvswitch > >> > >> Ben, Justin, can you help me fix all of this? > > > > Thanks for passing that along. > > > > A lot of these failures seem to involve unexpected timeouts. I wonder > > whether the buildds are so overloaded that some of the 10-second > > timeouts in the testsuite are just too short. Usually, this is a > > generous timeout interval. > > > > I sent a patch that should help to debug the problem by doing more logging: > > https://patchwork.ozlabs.org/patch/966087/ > > > > It won't help with tests that fully succeed, because the logs by default > > are discarded, but for tests that have a sequence of waits, in which one > > eventually fails, it will allow us to see how long the successful waits > > took. > > > > Any chance you could apply that patch and try another build? Feel free > > to wait for review, if you prefer. > > > > Hi, > > I've just uploaded OVS with that patch. Thanks, I think it's a very good > idea. And indeed, it looks like failing arch are the slower ones. I'm pretty pleased with the theory myself, but the results tend to show that it wasn't the problem. In most of the tests that eventually failed, the wait failure was preceded by other waits that succeeded immediately, and the longest wait I see is 3 seconds. I'll look for other possible causes. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
On 09/04/2018 11:06 PM, Ben Pfaff wrote: > On Tue, Sep 04, 2018 at 09:20:45AM +0200, Thomas Goirand wrote: >> On 09/02/2018 03:12 AM, Justin Pettit wrote: >>> On Sep 1, 2018, at 3:52 PM, Ben Pfaff wrote: On Sat, Sep 01, 2018 at 01:23:32PM -0700, Justin Pettit wrote: > >> On Sep 1, 2018, at 12:21 PM, Thomas Goirand wrote: >> >> >> The only one failure: >> >> 2633: ovn -- ACL rate-limited logging FAILED >> (ovn.at:6516) > > My guess if that this is meter-related. Can you send the ovs-vswitchd.log > and testsuite.log so I can take a look? It probably hasn't changed from what he sent the first time around. >>> >>> Yes, "testsuite.log" was in the original message, so I don't need that. >>> Thomas, can you send me "ovs-vswitchd.log" and "ovn-controller.log"? Does >>> it consistently fail for you? >>> >>> --Justin >> >> Hi, >> >> As I blacklisted the above test, I uploaded to Sid, and now there's a >> number of failures on non-intel arch: >> >> https://buildd.debian.org/status/package.php?p=openvswitch >> https://buildd.debian.org/status/logs.php?pkg=openvswitch >> >> Ben, Justin, can you help me fix all of this? > > Thanks for passing that along. > > A lot of these failures seem to involve unexpected timeouts. I wonder > whether the buildds are so overloaded that some of the 10-second > timeouts in the testsuite are just too short. Usually, this is a > generous timeout interval. > > I sent a patch that should help to debug the problem by doing more logging: > https://patchwork.ozlabs.org/patch/966087/ > > It won't help with tests that fully succeed, because the logs by default > are discarded, but for tests that have a sequence of waits, in which one > eventually fails, it will allow us to see how long the successful waits > took. > > Any chance you could apply that patch and try another build? Feel free > to wait for review, if you prefer. > Hi, I've just uploaded OVS with that patch. Thanks, I think it's a very good idea. And indeed, it looks like failing arch are the slower ones. Cheers, Thomas Goirand (zigo) ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
> On Sep 3, 2018, at 2:25 AM, Thomas Goirand wrote: > >> Does it consistently fail for you? > > It's very deterministically failing indeed. Thank you for the information. I understand the nature of the issue, but I'm still thinking about the best way to address it. In the test, we send three sets of 100 packets. One of the sets drops packets at a rate of 10 per second, one at a rate of 5 per second, and one not at all. On my computer, it takes roughly 0.67 seconds to send those 300 packets. It appears that your computer takes over 15 seconds, which means that the 10 per second meter isn't hitting at all, and all packets are getting through. The test was intended to allow some flexibility in run-time, but not that extreme. I hope to have a fix out for it tomorrow, though. --Justin ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
On Tue, Sep 04, 2018 at 09:20:45AM +0200, Thomas Goirand wrote: > On 09/02/2018 03:12 AM, Justin Pettit wrote: > > > >> On Sep 1, 2018, at 3:52 PM, Ben Pfaff wrote: > >> > >> On Sat, Sep 01, 2018 at 01:23:32PM -0700, Justin Pettit wrote: > >>> > On Sep 1, 2018, at 12:21 PM, Thomas Goirand wrote: > > > The only one failure: > > 2633: ovn -- ACL rate-limited logging FAILED > (ovn.at:6516) > >>> > >>> My guess if that this is meter-related. Can you send the ovs-vswitchd.log > >>> and testsuite.log so I can take a look? > >> > >> It probably hasn't changed from what he sent the first time around. > > > > Yes, "testsuite.log" was in the original message, so I don't need that. > > Thomas, can you send me "ovs-vswitchd.log" and "ovn-controller.log"? Does > > it consistently fail for you? > > > > --Justin > > Hi, > > As I blacklisted the above test, I uploaded to Sid, and now there's a > number of failures on non-intel arch: > > https://buildd.debian.org/status/package.php?p=openvswitch > https://buildd.debian.org/status/logs.php?pkg=openvswitch > > Ben, Justin, can you help me fix all of this? Thanks for passing that along. A lot of these failures seem to involve unexpected timeouts. I wonder whether the buildds are so overloaded that some of the 10-second timeouts in the testsuite are just too short. Usually, this is a generous timeout interval. I sent a patch that should help to debug the problem by doing more logging: https://patchwork.ozlabs.org/patch/966087/ It won't help with tests that fully succeed, because the logs by default are discarded, but for tests that have a sequence of waits, in which one eventually fails, it will allow us to see how long the successful waits took. Any chance you could apply that patch and try another build? Feel free to wait for review, if you prefer. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
On 09/02/2018 03:12 AM, Justin Pettit wrote: > >> On Sep 1, 2018, at 3:52 PM, Ben Pfaff wrote: >> >> On Sat, Sep 01, 2018 at 01:23:32PM -0700, Justin Pettit wrote: >>> On Sep 1, 2018, at 12:21 PM, Thomas Goirand wrote: The only one failure: 2633: ovn -- ACL rate-limited logging FAILED (ovn.at:6516) >>> >>> My guess if that this is meter-related. Can you send the ovs-vswitchd.log >>> and testsuite.log so I can take a look? >> >> It probably hasn't changed from what he sent the first time around. > > Yes, "testsuite.log" was in the original message, so I don't need that. > Thomas, can you send me "ovs-vswitchd.log" and "ovn-controller.log"? Does it > consistently fail for you? > > --Justin Hi, As I blacklisted the above test, I uploaded to Sid, and now there's a number of failures on non-intel arch: https://buildd.debian.org/status/package.php?p=openvswitch https://buildd.debian.org/status/logs.php?pkg=openvswitch Ben, Justin, can you help me fix all of this? While you do it, I will try to add DPDK support in the Debian package. Cheers, Thomas Goirand (zigo) ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
> On Sep 1, 2018, at 3:52 PM, Ben Pfaff wrote: > > On Sat, Sep 01, 2018 at 01:23:32PM -0700, Justin Pettit wrote: >> >>> On Sep 1, 2018, at 12:21 PM, Thomas Goirand wrote: >>> >>> >>> The only one failure: >>> >>> 2633: ovn -- ACL rate-limited logging FAILED (ovn.at:6516) >> >> My guess if that this is meter-related. Can you send the ovs-vswitchd.log >> and testsuite.log so I can take a look? > > It probably hasn't changed from what he sent the first time around. Yes, "testsuite.log" was in the original message, so I don't need that. Thomas, can you send me "ovs-vswitchd.log" and "ovn-controller.log"? Does it consistently fail for you? --Justin ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
On Sat, Sep 01, 2018 at 01:23:32PM -0700, Justin Pettit wrote: > > > On Sep 1, 2018, at 12:21 PM, Thomas Goirand wrote: > > > > > > The only one failure: > > > > 2633: ovn -- ACL rate-limited logging FAILED (ovn.at:6516) > > My guess if that this is meter-related. Can you send the ovs-vswitchd.log and > testsuite.log so I can take a look? It probably hasn't changed from what he sent the first time around. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
> On Sep 1, 2018, at 12:21 PM, Thomas Goirand wrote: > > > The only one failure: > > 2633: ovn -- ACL rate-limited logging FAILED (ovn.at:6516) My guess if that this is meter-related. Can you send the ovs-vswitchd.log and testsuite.log so I can take a look? Thanks, —Justin ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [openvswitch 2.10.0+2018.08.28+git.e0cea85314+ds2] testsuite: 975 2347 2482 2483 2633 failed
On 08/31/2018 10:13 PM, Ben Pfaff wrote:
> On Fri, Aug 31, 2018 at 12:11:20PM +0200, Thomas Goirand wrote:
>> Building OpenVSwitch tip of branch-2.10 (sha256 e0cea85314) produced
>> failures. The CA cert failures are probably related to OpenSSL 1.1.1,
>> currently in Sid. Please see https://bugs.debian.org/900153. For the
>> others, I don't know what's going on.
>
> When I test it myself, I do see problems related to key length. I
> backported commit 3391136cf7fa ("tests: Use the default key length when
> generating RSA keys") to help with that.
Indeed, that's the issue with OpenSSL 1.1.1, which has a longer minimum
key length. Thanks a lot for this backport!
> If you apply that, what's left?
The only one failure:
2633: ovn -- ACL rate-limited logging FAILED (ovn.at:6516)
Cheers,
Thomas Goirand (zigo)
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
