[OE-core] [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES
Now the test markup of the development tools tests is complete, this
can be further tweaked to auto run the correct tests.
Signed-off-by: Richard Purdie
---
meta/classes/testimage.bbclass | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index b00d47441b2..4e9494296e1 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?= "${WORKDIR}/testimage/packages"
TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
-BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi ptest dnf rpm
parselogs logrotate connman systemd oe_syslog pam stap ldd xorg"
-DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip buildgalculator "
+BASICTESTSUITE = "\
+ping date df ssh scp python perl apt opkg gi ptest dnf rpm parselogs \
+logrotate connman systemd oe_syslog pam stap ldd xorg kernelmodule gcc \
+buildcpio buildlzip buildgalculator"
DEFAULT_TEST_SUITES = "${BASICTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${BASICTESTSUITE}
${DEVTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${BASICTESTSUITE} ${DEVTESTSUITE}"
# aarch64 has no graphics
DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
--
2.17.1
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES
On 11/9/18 6:29 AM, Richard Purdie wrote:
> Now the test markup of the development tools tests is complete, this
> can be further tweaked to auto run the correct tests.
>
> Signed-off-by: Richard Purdie
> ---
> meta/classes/testimage.bbclass | 8
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
> index b00d47441b2..4e9494296e1 100644
> --- a/meta/classes/testimage.bbclass
> +++ b/meta/classes/testimage.bbclass
> @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?=
> "${WORKDIR}/testimage/packages"
> TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
> TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
>
> -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi ptest dnf rpm
> parselogs logrotate connman systemd oe_syslog pam stap ldd xorg"
> -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip buildgalculator "
> +BASICTESTSUITE = "\
> +ping date df ssh scp python perl apt opkg gi ptest dnf rpm parselogs \
> +logrotate connman systemd oe_syslog pam stap ldd xorg kernelmodule gcc \
> +buildcpio buildlzip buildgalculator"
Again Nak. You are forcing folks to have x11, systemd etc in the "Basic".
- armin
>
> DEFAULT_TEST_SUITES = "${BASICTESTSUITE}"
> -DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${BASICTESTSUITE}
> ${DEVTESTSUITE}"
> -DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${BASICTESTSUITE}
> ${DEVTESTSUITE}"
>
> # aarch64 has no graphics
> DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES
On Sun, 2018-11-11 at 12:52 -0800, akuster808 wrote:
> On 11/9/18 6:29 AM, Richard Purdie wrote:
> > Now the test markup of the development tools tests is complete,
> > this
> > can be further tweaked to auto run the correct tests.
> >
> > Signed-off-by: Richard Purdie
> > ---
> > meta/classes/testimage.bbclass | 8
> > 1 file changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/meta/classes/testimage.bbclass
> > b/meta/classes/testimage.bbclass
> > index b00d47441b2..4e9494296e1 100644
> > --- a/meta/classes/testimage.bbclass
> > +++ b/meta/classes/testimage.bbclass
> > @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?=
> > "${WORKDIR}/testimage/packages"
> > TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
> > TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
> >
> > -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi
> > ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam
> > stap ldd xorg"
> > -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip
> > buildgalculator "
> > +BASICTESTSUITE = "\
> > +ping date df ssh scp python perl apt opkg gi ptest dnf rpm
> > parselogs \
> > +logrotate connman systemd oe_syslog pam stap ldd xorg
> > kernelmodule gcc \
> > +buildcpio buildlzip buildgalculator"
>
> Again Nak. You are forcing folks to have x11, systemd etc in the
> "Basic".
No, I'm not. If the image doesn't have x11 in it, the test will show
"skipped". If the image doesn't have systemd in it, it will show those
tests as "skipped". If you just want to test ptest, you can specify
that, which is what you had to do before this change anyway *just* to
run ptest.
So I'm not really seeing the problem?
The only risk with doing this is that a test may be skipped when it
shouldn't be. The plan is to verify the right set of test results were
obtained when the QA overall results from a release build are compared
against a known list of tests which should have run.
Cheers,
Richard
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES
On 11/11/18 1:18 PM, richard.pur...@linuxfoundation.org wrote:
> On Sun, 2018-11-11 at 12:52 -0800, akuster808 wrote:
>> On 11/9/18 6:29 AM, Richard Purdie wrote:
>>> Now the test markup of the development tools tests is complete,
>>> this
>>> can be further tweaked to auto run the correct tests.
>>>
>>> Signed-off-by: Richard Purdie
>>> ---
>>> meta/classes/testimage.bbclass | 8
>>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/meta/classes/testimage.bbclass
>>> b/meta/classes/testimage.bbclass
>>> index b00d47441b2..4e9494296e1 100644
>>> --- a/meta/classes/testimage.bbclass
>>> +++ b/meta/classes/testimage.bbclass
>>> @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?=
>>> "${WORKDIR}/testimage/packages"
>>> TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
>>> TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
>>>
>>> -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi
>>> ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam
>>> stap ldd xorg"
>>> -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip
>>> buildgalculator "
>>> +BASICTESTSUITE = "\
>>> +ping date df ssh scp python perl apt opkg gi ptest dnf rpm
>>> parselogs \
>>> +logrotate connman systemd oe_syslog pam stap ldd xorg
>>> kernelmodule gcc \
>>> +buildcpio buildlzip buildgalculator"
>> Again Nak. You are forcing folks to have x11, systemd etc in the
>> "Basic".
> No, I'm not. If the image doesn't have x11 in it, the test will show
> "skipped". If the image doesn't have systemd in it, it will show those
> tests as "skipped". If you just want to test ptest, you can specify
> that, which is what you had to do before this change anyway *just* to
> run ptest.
>
> So I'm not really seeing the problem?
So for meta-security, since it needs python and perl, will it re-run
those tests every time I run the meta-security ptests? if it does, one
may interpret those changes as a negative to the overall testing experience.
- armin
>
> The only risk with doing this is that a test may be skipped when it
> shouldn't be. The plan is to verify the right set of test results were
> obtained when the QA overall results from a release build are compared
> against a known list of tests which should have run.
>
> Cheers,
>
> Richard
>
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 11/12] testimage: Further cleanup DEFAULT_TEST_SUITES
On Sun, 2018-11-11 at 13:34 -0800, akuster808 wrote:
> On 11/11/18 1:18 PM, richard.pur...@linuxfoundation.org wrote:
> > On Sun, 2018-11-11 at 12:52 -0800, akuster808 wrote:
> > > On 11/9/18 6:29 AM, Richard Purdie wrote:
> > > > Now the test markup of the development tools tests is complete,
> > > > this
> > > > can be further tweaked to auto run the correct tests.
> > > >
> > > > Signed-off-by: Richard Purdie <
> > > > richard.pur...@linuxfoundation.org>
> > > > ---
> > > > meta/classes/testimage.bbclass | 8
> > > > 1 file changed, 4 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/meta/classes/testimage.bbclass
> > > > b/meta/classes/testimage.bbclass
> > > > index b00d47441b2..4e9494296e1 100644
> > > > --- a/meta/classes/testimage.bbclass
> > > > +++ b/meta/classes/testimage.bbclass
> > > > @@ -40,12 +40,12 @@ TEST_NEEDED_PACKAGES_DIR ?=
> > > > "${WORKDIR}/testimage/packages"
> > > > TEST_EXTRACTED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/extracted"
> > > > TEST_PACKAGED_DIR ?= "${TEST_NEEDED_PACKAGES_DIR}/packaged"
> > > >
> > > > -BASICTESTSUITE = "ping date df ssh scp python perl apt opkg gi
> > > > ptest dnf rpm parselogs logrotate connman systemd oe_syslog pam
> > > > stap ldd xorg"
> > > > -DEVTESTSUITE = "gcc kernelmodule buildcpio buildlzip
> > > > buildgalculator "
> > > > +BASICTESTSUITE = "\
> > > > +ping date df ssh scp python perl apt opkg gi ptest dnf rpm
> > > > parselogs \
> > > > +logrotate connman systemd oe_syslog pam stap ldd xorg
> > > > kernelmodule gcc \
> > > > +buildcpio buildlzip buildgalculator"
> > >
> > > Again Nak. You are forcing folks to have x11, systemd etc in the
> > > "Basic".
> >
> > No, I'm not. If the image doesn't have x11 in it, the test will
> > show
> > "skipped". If the image doesn't have systemd in it, it will show
> > those
> > tests as "skipped". If you just want to test ptest, you can specify
> > that, which is what you had to do before this change anyway *just*
> > to
> > run ptest.
> >
> > So I'm not really seeing the problem?
>
> So for meta-security, since it needs python and perl, will it re-run
> those tests every time I run the meta-security ptests? if it does,
> one may interpret those changes as a negative to the overall testing
> experience.
It depends. It will probably continue to do what it did before which is
to "auto" run tests, unless you're building one of the core-image-XXX
images which had specific tests defined.
If you want to just run the ptest tests, just set:
TEST_SUITES = "ptest"
There are likely some "testing experience" issues but I think these
changes are a key step in improving it, by making the tests much more
automatic without having to define specific test suites for specific
images.
You can still specify specific test subsets just as you've always been
able to.
I agree the output from testing is confusing and I have some ideas on
how to improve that too, but one step (patch) at a time.
Cheers,
Richard
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
