This series adds support to execute the gnu test suites for binutils, gcc and glibc. With the intention for enabling automated test running of these test suites within the OEQA framework such that they can be executed by the Yocto Autobuilder.
The test suites covered need significant resources or build artifacts such that running them on the target is undesirable which rules out the use of ptest. Because of this the test suites can be run on the build host and call out to the target for execution (via ssh or using qemu usermode). The following implementation adds a do_check task to gcc-runtime, and creates recipes for binutils and glibc in order to execute the test suite on the build host. For binutils the reason for a separate recipe is due to test suite dependence on target libraries (libgcc, glibc) and the issues associated with a -cross recipe depending on target libraries. For glibc the reason for a recipe is due to the dependency chain (libgcc -> glibc -> libgcc-initial) and the test suite dependence on libgcc. Target execution is another important issue specifically since target execution is slow to extremely slow depending on the physical or emulated target performance. In order to provide faster execution performance qemu linux-user is implemented alongside qemu system emulation (via ssh+nfs). In initial testing qemu linux user vs qemu system emulation provided performance gains of between 10x to 60x whilst initially having small pass/fail differences. Further work as covered in this series reduces the pass/fail differences for gcc/gcc-runtime close to 0, which can be further reduced by marking known failures with test result filtering. However glibc is more strict with its expectations of CPU implementation as well as syscall behaviour. Additionally glibc expects to be able to execute OS tools such as 'sh' and 'echo' of which is not easy to provide in the recipe-sysroot (bindir is not populated into the sysroot). As such correct test results for glibc rely on execution with qemu system emulation or on a physical target. This series however still includes qemu user execution for glibc which can be useful (especially on slow targets) assuming the known failing tests are excluded. This series also introduces some OEQA test cases which cover running the test suites. The test cases are split into binutils, gcc and glibc. Individual test cases provide execution of the sub-suites of tests within each target. For example binutils has binutils, gas and ld and suites which can each be executed independently. The test cases populate the individual test suite test cases into testresults similar to ptestresults so that resulttool can analyse them. Filtering of the failed test cases where expected failures occur is not included in this series. The OEQA test cases implement execution on qemu linux user by default. Subclasses implement qemu system emulation setup and configuration for running the tests. Changes in v2: - Dropped merged patches from series: - scripts/lib/resulttool/report.py: Add more result types - oeqa/utils/nfs: Add unfs_server function to setup a userspace NFS server - binutils: Fix mips patch which changes default emulation - Added dejagnu recipe to oe-core, imported from meta-oe - Changes from binutils-cross to binutils-cross-testsuite recipe - Dropped gold and libiberty test suite execution - Changes from gcc-cross to only gcc-runtime for gcc/g++ tests - BUILD_TEST_* to TOOLCHAIN_TEST_* - New changes to oeqa/core for tagging tests, and patches to fix up broken tests in the oeqa/core testsuite - New changes to oeqa/selftest for '--run-only-tags' and '--run-exclude-tags' - Changes to new selftest tests to populate results like ptestresults - Mark new selftest tests with 'machine' tag --- Nathan Rossi (12): dejagnu: Add dejagnu for binutils/gcc test suites binutils-cross-testsuite: Create recipe for test suite execution gcc-runtime: Add do_check task for executing gcc test suites gcc-common.inc: Process staging fixme with correct target/native sysroot glibc-testsuite: Create a recipe to implement glibc test suite oeqa/core/tests: Skip test_fail_duplicated_module oeqa/core/tests: Fix test_data module tests oeqa/core: Rework OETestTag and remove unused OETestFilter oeqa/selftest: Add test run filtering based on test tags oeqa/selftest/binutils: Create selftest case for binutils test suite oeqa/selftest/gcc: Create selftest case for gcc test suite oeqa/selftest/glibc: Create selftest case for glibc test suite meta/lib/oeqa/core/context.py | 4 +- meta/lib/oeqa/core/decorator/__init__.py | 20 ++-- meta/lib/oeqa/core/decorator/oetag.py | 27 ----- meta/lib/oeqa/core/loader.py | 61 +++------- meta/lib/oeqa/core/tests/cases/data.py | 2 +- meta/lib/oeqa/core/tests/cases/oetag.py | 21 +++- meta/lib/oeqa/core/tests/common.py | 4 +- meta/lib/oeqa/core/tests/test_data.py | 10 +- meta/lib/oeqa/core/tests/test_decorators.py | 77 +++++++----- meta/lib/oeqa/core/tests/test_loader.py | 26 +--- meta/lib/oeqa/selftest/cases/binutils.py | 64 ++++++++++ meta/lib/oeqa/selftest/cases/gcc.py | 111 ++++++++++++++++++ meta/lib/oeqa/selftest/cases/glibc.py | 97 +++++++++++++++ meta/lib/oeqa/selftest/context.py | 11 ++ .../glibc/glibc-testsuite_2.30.bb | 51 ++++++++ .../glibc/glibc/check-test-wrapper | 71 +++++++++++ .../binutils/binutils-cross-testsuite_2.32.bb | 83 +++++++++++++ .../recipes-devtools/dejagnu/dejagnu_1.6.2.bb | 15 +++ meta/recipes-devtools/gcc/gcc-common.inc | 2 +- meta/recipes-devtools/gcc/gcc-runtime.inc | 42 +++++++ meta/recipes-devtools/gcc/gcc-testsuite.inc | 106 +++++++++++++++++ 21 files changed, 762 insertions(+), 143 deletions(-) delete mode 100644 meta/lib/oeqa/core/decorator/oetag.py create mode 100644 meta/lib/oeqa/selftest/cases/binutils.py create mode 100644 meta/lib/oeqa/selftest/cases/gcc.py create mode 100644 meta/lib/oeqa/selftest/cases/glibc.py create mode 100644 meta/recipes-core/glibc/glibc-testsuite_2.30.bb create mode 100644 meta/recipes-core/glibc/glibc/check-test-wrapper create mode 100644 meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb create mode 100644 meta/recipes-devtools/dejagnu/dejagnu_1.6.2.bb create mode 100644 meta/recipes-devtools/gcc/gcc-testsuite.inc --- 2.23.0.rc1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core