[OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution
From: Leonardo Sandoval
The main idea is to isolate the oe-selftest execution so neither the current
build dir nor the configuration data is touch/polluted. This approach uses
a wrapper script (which is the one presented on this commit) which creates
a unique directory and inside it copies all scripts and metadata, re-initializes
the enviroment (re-sources oe-init-build-env) and finally launches
the oe-selftest-internal (which used to be oe-selftest) command, passing command
arguments to the latter.
The new build directory created when re-initializing the enviroment has the
same configuration data (local.conf, auto.conf, site.conf) as the
main build directory. The latter means that one can set DL_DIR and SSTATE_DIR
into /conf/site.conf and the oe-selftest execution will use
this.
[YOCTO #11429]
Signed-off-by: Leonardo Sandoval
---
scripts/oe-selftest | 108 ---
scripts/oe-selftest-internal | 75 ++
2 files changed, 135 insertions(+), 48 deletions(-)
create mode 100755 scripts/oe-selftest-internal
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index 1bf860a415..0505f943e4 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -1,5 +1,7 @@
-#!/usr/bin/env python3
+#!/bin/sh
+# scripts/oe-selftest: calls oe-selftest-internal in a isolated environment
+#
# Copyright (c) 2013-2017 Intel Corporation
#
# This program is free software; you can redistribute it and/or modify
@@ -14,62 +16,72 @@
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+# Before anything else, check oe-core environment
+if [ -z "$BUILDDIR" ]; then
+echo "Please initialize the OE-Core environment through oe-init-build-env
script"
+exit 1
+fi
+
+# Variable definitions
+ORIGBUILDDIR="$BUILDDIR"
+OESELFTESTSCRIPTDIR="$(which oe-selftest)"
+SCRIPTSDIR="$(dirname $OESELFTESTSCRIPTDIR)"
+OECOREDIR="$(dirname $SCRIPTSDIR)"
-# DESCRIPTION
-# This script runs tests defined in meta/lib/oeqa/selftest/
-# It's purpose is to automate the testing of different bitbake tools.
-# To use it you just need to source your build environment setup script and
-# add the meta-selftest layer to your BBLAYERS.
-# Call the script as: "oe-selftest -a" to run all the tests in
meta/lib/oeqa/selftest/
-# Call the script as: "oe-selftest -r .." to run just a
single test
-# E.g: "oe-selftest -r bblayers.BitbakeLayers" will run just the BitbakeLayers
class from meta/lib/oeqa/selftest/bblayers.py
+# oe-selftest-$$ related
+OESELFTESTDIR="$ORIGBUILDDIR/oe-selftest-$$"
+OESELFTESTBUILDDIR="$OESELFTESTDIR/build"
+# Return immediately in case no test execution
+ADDTESTS="$(echo "$@" | grep -i '\-a')"
+SOMETESTS="$(echo "$@" | grep -i '\-r')"
+if [ -z "$ADDTESTS" -a -z "$SOMETESTS" ]; then
+if [ -z "$@" ]; then
+ oe-selftest-internal -h
+else
+ oe-selftest-internal "$@"
+fi
+exit 0
+fi
+# Create directories (build, build/conf and layers)
+mkdir -p $OESELFTESTDIR $OESELFTESTBUILDDIR/conf $OESELFTESTLAYERSDIR
-import os
-import sys
-import argparse
-import logging
+# Populate the new build/conf, bblayers.conf will be touch latter to reflect
new layer structure
+cp $ORIGBUILDDIR/conf/* $OESELFTESTBUILDDIR/conf
-scripts_path = os.path.dirname(os.path.realpath(__file__))
-lib_path = scripts_path + '/lib'
-sys.path = sys.path + [lib_path]
-import argparse_oe
-import scriptutils
-import scriptpath
-scriptpath.add_oe_lib_path()
-scriptpath.add_bitbake_lib_path()
+# Copy OE-Core meta directories
+cp -a $OECOREDIR/meta-selftest $OESELFTESTDIR
+cp -a $OECOREDIR/meta-skeleton $OESELFTESTDIR
+cp -a $OECOREDIR/meta-yocto-bsp $OESELFTESTDIR
-from oeqa.utils import load_test_components
-from oeqa.core.exception import OEQAPreRun
+# Copy non-metadata files
+cp -a $OECOREDIR/bitbake$OESELFTESTDIR
+cp -a $SCRIPTSDIR $OESELFTESTDIR
+cp $OECOREDIR/oe-init-build-env $OESELFTESTDIR
+cp $OECOREDIR/.templateconf $OESELFTESTDIR
-logger = scriptutils.logger_create('oe-selftest', stream=sys.stdout)
+# Copy all layers define in BBLAYERS and construct the new bblayers.conf at
the same time
+BBLAYERS="$(bitbake -e | grep ^BBLAYERS= | sed -e s/BBLAYERS=// -e s/\"//g)"
+for src in $BBLAYERS; do
+cp -a $src $OESELFTESTDIR
+# rename the new layer into the new bblayers.conf
+tgt="$OESELFTESTDIR/$(basename $src)"
+sed -e "s|$src|$tgt|" -i $OESELFTESTBUILDDIR/conf/bblayers.conf
+done
-def main():
-description = "Script that runs unit tests against bitbake and other Yocto
related tools. The goal is to validate tools functionality and metadata
integrity. Refer to https://wiki.yoctoproject.org/wiki/Oe-selftest for more
information."
-parser = argparse_oe.ArgumentParser(description=description)
+# Reinitialized environment with new metadata a
Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution
On Mon, 2017-11-13 at 10:17 -0800, leonardo.sandoval.gonza...@linux.intel.com wrote: > From: Leonardo Sandoval > > The main idea is to isolate the oe-selftest execution so neither the > current build dir nor the configuration data is touch/polluted. This > approach uses a wrapper script (which is the one presented on this > commit) which creates a unique directory and inside it copies all > scripts and metadata, re-initializes the enviroment (re-sources oe- > init-build-env) and finally launches the oe-selftest-internal (which > used to be oe-selftest) command, passing command arguments to the > latter. This mode of operation may or may not be desirable. Can it be made optional? In refkit CI testing, several selftests run tests on build artifacts (primarily the images) produced during the previous build and only build them if needed, i.e. they run "bitbake some-image" and that is usually fast because the image already exists. Reusing sstate and download dir also is important for speed. Forcing all tests to run in a clean environment would make the overall CI run slower. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution
On Tue, 14 Nov 2017 09:24:30 +0100 Patrick Ohly wrote: > On Mon, 2017-11-13 at 10:17 -0800, > leonardo.sandoval.gonza...@linux.intel.com wrote: > > From: Leonardo Sandoval > > > > The main idea is to isolate the oe-selftest execution so neither the > > current build dir nor the configuration data is touch/polluted. This > > approach uses a wrapper script (which is the one presented on this > > commit) which creates a unique directory and inside it copies all > > scripts and metadata, re-initializes the enviroment (re-sources oe- > > init-build-env) and finally launches the oe-selftest-internal (which > > used to be oe-selftest) command, passing command arguments to the > > latter. > > This mode of operation may or may not be desirable. Can it be made > optional? good idea. However, you can call the oe-selftest-internal for the this purpose. > > In refkit CI testing, several selftests run tests on build artifacts > (primarily the images) produced during the previous build and only > build them if needed, i.e. they run "bitbake some-image" and that is > usually fast because the image already exists. Reusing sstate and > download dir also is important for speed. oe-selftest creates a new build/conf folder, with the same conf files as the ones present in your current build/conf, so this means that you can set there DL_DIR and SSTATE_DIR (for example) on your main local.conf and these will be used by oe-selftest, effectively reusing these folders. > > Forcing all tests to run in a clean environment would make the overall > CI run slower. Agree. better to start with a 'hot cache' scenario, and this can be accomplished the way I mentioned before. > > -- > Best Regards, Patrick Ohly > > The content of this message is my personal opinion only and although > I am an employee of Intel, the statements I make here in no way > represent Intel's position on the issue, nor am I authorized to speak > on behalf of Intel on this matter. > > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution
On Tue, 2017-11-14 at 10:09 -0600, Leonardo Sandoval wrote: > On Tue, 14 Nov 2017 09:24:30 +0100 > Patrick Ohly wrote: > > > On Mon, 2017-11-13 at 10:17 -0800, > > leonardo.sandoval.gonza...@linux.intel.com wrote: > > > From: Leonardo Sandoval > > om> > > > > > > The main idea is to isolate the oe-selftest execution so neither > > > the > > > current build dir nor the configuration data is touch/polluted. > > > This > > > approach uses a wrapper script (which is the one presented on > > > this > > > commit) which creates a unique directory and inside it copies all > > > scripts and metadata, re-initializes the enviroment (re-sources > > > oe- > > > init-build-env) and finally launches the oe-selftest-internal > > > (which > > > used to be oe-selftest) command, passing command arguments to the > > > latter. > > > > This mode of operation may or may not be desirable. Can it be made > > optional? > > good idea. However, you can call the oe-selftest-internal for the > this purpose. While doable, it feels wrong to rename something as "internal" and then still expect people to call it directly. A command line parameter for the new oe-selftest which controls this behavior and just calls oe-selftest-internal directly when requested feels cleaner and more discoverable. Is oe-selftest-internal even going to be in the default PATH? It probably shouldn't be, once it truly becomes an implementation detail. > > In refkit CI testing, several selftests run tests on build > > artifacts > > (primarily the images) produced during the previous build and only > > build them if needed, i.e. they run "bitbake some-image" and that > > is > > usually fast because the image already exists. Reusing sstate and > > download dir also is important for speed. > > oe-selftest creates a new build/conf folder, with the same conf files > as the ones present in your current build/conf, so this means that > you can set there DL_DIR and SSTATE_DIR (for example) on your main > local.conf and these will be used by oe-selftest, effectively reusing > these folders. Instead of leaving it to the developer to figure this out, should the oe-selftest have --[no-]reuse-download and --[no]-reuse-sstate options? > > -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution
On Tue, 14 Nov 2017 17:09:28 +0100 Patrick Ohly wrote: > On Tue, 2017-11-14 at 10:09 -0600, Leonardo Sandoval wrote: > > On Tue, 14 Nov 2017 09:24:30 +0100 > > Patrick Ohly wrote: > > > > > On Mon, 2017-11-13 at 10:17 -0800, > > > leonardo.sandoval.gonza...@linux.intel.com wrote: > > > > From: Leonardo Sandoval > > > om> > > > > > > > > The main idea is to isolate the oe-selftest execution so neither > > > > the > > > > current build dir nor the configuration data is touch/polluted. > > > > This > > > > approach uses a wrapper script (which is the one presented on > > > > this > > > > commit) which creates a unique directory and inside it copies all > > > > scripts and metadata, re-initializes the enviroment (re-sources > > > > oe- > > > > init-build-env) and finally launches the oe-selftest-internal > > > > (which > > > > used to be oe-selftest) command, passing command arguments to the > > > > latter. > > > > > > This mode of operation may or may not be desirable. Can it be made > > > optional? > > > > good idea. However, you can call the oe-selftest-internal for the > > this purpose. > > While doable, it feels wrong to rename something as "internal" and then > still expect people to call it directly. > > A command line parameter for the new oe-selftest which controls this > behavior and just calls oe-selftest-internal directly when requested > feels cleaner and more discoverable. > > Is oe-selftest-internal even going to be in the default PATH? It > probably shouldn't be, once it truly becomes an implementation detail. where can we placed it besides the scripts folder? > > > > In refkit CI testing, several selftests run tests on build > > > artifacts > > > (primarily the images) produced during the previous build and only > > > build them if needed, i.e. they run "bitbake some-image" and that > > > is > > > usually fast because the image already exists. Reusing sstate and > > > download dir also is important for speed. > > > > oe-selftest creates a new build/conf folder, with the same conf files > > as the ones present in your current build/conf, so this means that > > you can set there DL_DIR and SSTATE_DIR (for example) on your main > > local.conf and these will be used by oe-selftest, effectively reusing > > these folders. > > Instead of leaving it to the developer to figure this out, should the > oe-selftest have --[no-]reuse-download and --[no]-reuse-sstate options? sounds good. By default, we will let reuse download/sstate. Good input. I will send a v3. > > > > > -- > Best Regards, Patrick Ohly > > The content of this message is my personal opinion only and although > I am an employee of Intel, the statements I make here in no way > represent Intel's position on the issue, nor am I authorized to speak > on behalf of Intel on this matter. > > -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution
On Tue, 2017-11-14 at 11:42 -0600, Leonardo Sandoval wrote: > On Tue, 14 Nov 2017 17:09:28 +0100 > Patrick Ohly wrote: > > Is oe-selftest-internal even going to be in the default PATH? It > > probably shouldn't be, once it truly becomes an implementation > > detail. > > where can we placed it besides the scripts folder? scripts/lib/ perhaps, then call it from oe-selftest with an absolute path? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
