[Bug 1966017] Re: enable upstream python testsuite in autopkgtests
Attaching the updated debdiff in case we do decide we want this (even in the broken state) ** Patch added: "nftables_1.0.2-1ubuntu2.debdiff" https://bugs.launchpad.net/ubuntu/+source/nftables/+bug/1966017/+attachment/5572129/+files/nftables_1.0.2-1ubuntu2.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1966017 Title: enable upstream python testsuite in autopkgtests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nftables/+bug/1966017/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1966017] Re: enable upstream python testsuite in autopkgtests
Turns out I wasn't looking closely enough at the autopkgtest logs from my local testing - I was trying to run the tests with the pre-built binary packages and whilst it would indicate the internaltest-py.sh tests were passing, they were actually completely failing without any error indication: autopkgtest [19:10:18]: test internaltest-py.sh: [--- The nftables library at 'src/.libs/libnftables.so.1' does not exist. You need to build the project. autopkgtest [19:10:19]: test internaltest-py.sh: ---] autopkgtest [19:10:20]: test internaltest-py.sh: - - - - - - - - - - results - - - - - - - - - - internaltest-py.sh PASS So I think clearly this is a lot less straightforward than I originally thought and perhaps we shouldn't bother with it at this stage - also since as you say since it doesn't fail the autopkgtest run itself then is there any point trying to enable these tests as we won't be able to use it to easily detect regressions in that case. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1966017 Title: enable upstream python testsuite in autopkgtests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nftables/+bug/1966017/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1966017] Re: enable upstream python testsuite in autopkgtests
(If this were a build time testsuite, our log comparison process would pick up changes. We could *maybe* do something akin to how we try to detect new failing tests in openjdk in qrt's notes_testing/openjdk/ where we maybe compare our current adt runs of nftables against a prior run, and look for differences.) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1966017 Title: enable upstream python testsuite in autopkgtests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nftables/+bug/1966017/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1966017] Re: enable upstream python testsuite in autopkgtests
So this looks okay, there are unfortunately a bunch of errors in the tests with v1.0.2 against a 5.15 kernel because the 'egress' hook support was only added in 5.16 (https://git.kernel.org/linus/42df6e1d221dddc0f2acf2be37e68d553ad65f96). This results in the following output in a jammy VM: 96 test files, 69 files passed, 1889 unit tests, 717 error, 0 warning I'd like to suggest one additional change, adding a `-f` option to the nft-test.py invocation in debian/tests/internaltest-py.sh . From the README in test/py/ that tells the testsuite to carry on testing additional families in case of error. With that change in place, I get the following total results, again in a jammy VM with a 5.15 kernel: 96 test files, 69 files passed, 1889 unit tests, 5776 total executed, 719 error, 0 warning I have not run this in adt, but running the tests manually doesn't return an error value on exit(), even with 700+ errors, so I'm not sure how the adt test will detect a regression that causes more test cases to error. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1966017 Title: enable upstream python testsuite in autopkgtests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nftables/+bug/1966017/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1966017] Re: enable upstream python testsuite in autopkgtests
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1966017 Title: enable upstream python testsuite in autopkgtests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nftables/+bug/1966017/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1966017] Re: enable upstream python testsuite in autopkgtests
** Patch added: "nftables_1.0.2-1ubuntu2.debdiff" https://bugs.launchpad.net/ubuntu/+source/nftables/+bug/1966017/+attachment/5572061/+files/nftables_1.0.2-1ubuntu2.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1966017 Title: enable upstream python testsuite in autopkgtests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nftables/+bug/1966017/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
