Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi RP, Noted, thank you once again for your great help and inputs! Really glad to hear that resulttool was ready! We shall plan forward for future improvement in html reports and graphs. Also we shall look into future test case development if needed. Cheers, Yeoh Ee Peng -Original Message- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Thursday, February 21, 2019 5:44 AM To: Yeoh, Ee Peng ; openembedded-core@lists.openembedded.org Cc: Burton, Ross ; Eggleton, Paul Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt Hi Ee Peng, On Wed, 2019-02-20 at 06:27 +, Yeoh, Ee Peng wrote: > Thank you very much for all your help and inputs! > Would you like us to take all the improvements from your branch to > merge or squash with the base patchset and move forward with the one > remaining improvement below? I've done some further work on this today and the good news is I was able to sort out the git repo handling pieces and fix the test cases. With two of the test cases I ended up removing them as I've changed the functionality enough that they'd need to be rewritten. I've sent out a patch on top of your original work as well as a second patch to move some functionality into library functions to allow us to use it from the new code. I think this combination of patches should now be ready to merge. There will be fixes and improvements on top of this, e.g. I'd love to get some html reports and graphs but those are things that come later. The next step once this is merged is to start storing autobuilder test result data, generating reports and regression reports automatically from each test run. Its great to see this all coming together! Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi RP, Noted, thank you once again for your help and inputs! Really glad to hear that resulttool was ready! We shall plan forward for future improvement in html reports and graphs. Also we shall look into future test case development if needed. Cheers, Yeoh Ee Peng -Original Message- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Thursday, February 21, 2019 5:44 AM To: Yeoh, Ee Peng ; openembedded-core@lists.openembedded.org Cc: Burton, Ross ; Eggleton, Paul Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt Hi Ee Peng, On Wed, 2019-02-20 at 06:27 +, Yeoh, Ee Peng wrote: > Thank you very much for all your help and inputs! > Would you like us to take all the improvements from your branch to > merge or squash with the base patchset and move forward with the one > remaining improvement below? I've done some further work on this today and the good news is I was able to sort out the git repo handling pieces and fix the test cases. With two of the test cases I ended up removing them as I've changed the functionality enough that they'd need to be rewritten. I've sent out a patch on top of your original work as well as a second patch to move some functionality into library functions to allow us to use it from the new code. I think this combination of patches should now be ready to merge. There will be fixes and improvements on top of this, e.g. I'd love to get some html reports and graphs but those are things that come later. The next step once this is merged is to start storing autobuilder test result data, generating reports and regression reports automatically from each test run. Its great to see this all coming together! Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi Ee Peng, On Wed, 2019-02-20 at 06:27 +, Yeoh, Ee Peng wrote: > Thank you very much for all your help and inputs! > Would you like us to take all the improvements from your branch to > merge or squash with the base patchset and move forward with the one > remaining improvement below? I've done some further work on this today and the good news is I was able to sort out the git repo handling pieces and fix the test cases. With two of the test cases I ended up removing them as I've changed the functionality enough that they'd need to be rewritten. I've sent out a patch on top of your original work as well as a second patch to move some functionality into library functions to allow us to use it from the new code. I think this combination of patches should now be ready to merge. There will be fixes and improvements on top of this, e.g. I'd love to get some html reports and graphs but those are things that come later. The next step once this is merged is to start storing autobuilder test result data, generating reports and regression reports automatically from each test run. Its great to see this all coming together! Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi RP, Thank you very much for all your help and inputs! Would you like us to take all the improvements from your branch to merge or squash with the base patchset and move forward with the one remaining improvement below? > > * Revisit and redo the way the git branch handling is happening. > > We > > really want to model how oe-build-perf-report handles git repos > > for > > comparisons: > > - Its able to query data from git repos without changing the > > current > > working branch, > > - it can search on tag formats to find comparison data Best regards, Yeoh Ee Peng -Original Message- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Monday, February 18, 2019 6:46 AM To: Yeoh, Ee Peng ; openembedded-core@lists.openembedded.org Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt On Sun, 2019-02-17 at 17:54 +, Richard Purdie wrote: > > Despite my changes there are things that still need to be done. > > Essential things which need to happen before this code merges: > > > > * oe-git-archive is importing using the commit/branch of the current > > repo, not the data in the results file. Also now fixed. I put my patches into master-next too. With this working, I was able to run something along the lines of: for D in $1/*; do resulttool store $D $2 --allow-empty done on the autobuilder's recent results which lead to the creation of this repository: http://git.yoctoproject.org/cgit.cgi/yocto-testresults/ > > * Revisit and redo the way the git branch handling is happening. > > We > > really want to model how oe-build-perf-report handles git repos > > for > > comparisons: > > - Its able to query data from git repos without changing the > > current > > working branch, > > - it can search on tag formats to find comparison data Which means we now need to make the git branch functionality of the report and regression commands compare with the above repo, so we're a step closer to getting thie merged. Ultimately we'll auto-populate the above repo by having the autobuilder run a "store" command at the end of its runs. I have a feeling I may have broken the resulttool selftests so that is something else which will need to be fixed before anything merges. Time for me to step away from the keyboard for a bit too. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
RP, Noted, thanks. Cheers, Ee Peng -Original Message- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Monday, February 18, 2019 6:12 PM To: Yeoh, Ee Peng ; openembedded-core@lists.openembedded.org Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt On Mon, 2019-02-18 at 09:20 +, Yeoh, Ee Peng wrote: > Thank you for sharing on the selftest comparison consideration! > > I agreed with you that in the high level, selftest should be > independent of which HOST_DISTRO, it shall compared 2 selftest even > when the host distro are different. > > But in the case that the build have multiple set of selftest each with > slightly different environments (eg. host distro), in that case, will > it better to compare selftest more closely if possible with same host > distro used? In an ideal world, yes. In reality trying to do that and making it conditional will complicate the code for little "real" end difference though? Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
On Mon, 2019-02-18 at 09:20 +, Yeoh, Ee Peng wrote: > Thank you for sharing on the selftest comparison consideration! > > I agreed with you that in the high level, selftest should be > independent of which HOST_DISTRO, it shall compared 2 selftest even > when the host distro are different. > > But in the case that the build have multiple set of selftest each > with slightly different environments (eg. host distro), in that case, > will it better to compare selftest more closely if possible with same > host distro used? In an ideal world, yes. In reality trying to do that and making it conditional will complicate the code for little "real" end difference though? Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi Richard,
Thank you for sharing on the selftest comparison consideration!
I agreed with you that in the high level, selftest should be independent of
which HOST_DISTRO, it shall compared 2 selftest even when the host distro are
different.
But in the case that the build have multiple set of selftest each with slightly
different environments (eg. host distro), in that case, will it better to
compare selftest more closely if possible with same host distro used?
Cheers,
Ee Peng
-Original Message-
From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org]
Sent: Monday, February 18, 2019 5:08 PM
To: Yeoh, Ee Peng ;
openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi Ee Peng,
On Mon, 2019-02-18 at 08:09 +, Yeoh, Ee Peng wrote:
> I did some testing with the latest from resulttool: Update to use
> gitarchive library function.
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2
> 22&id=b9eecaabe56db5bcafff31e67cdabadc42e2d2e4
>
> I had 2 questions.
> 1. For "resulttool regression", currently it was comparing result id
> set without comprehending the difference in the host distro used to
> executed the oeselftest. Example: it was matching oeselftest run with
> fedora28 host distro with oeselftest run with ubuntu18 host distro, is
> this the expected behavior?
> Match: oeselftest_fedora-28_qemux86-64_20190201181656
>oeselftest_ubuntu-18.04_qemux86-64_20190201175023
> Match: oeselftest_fedora-26_qemux86-64_20190131144317
>oeselftest_fedora-26_qemux86-64_20190131144317
> Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
>oeselftest_fedora-28_qemux86-64_20190201181656
> Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
>oeselftest_opensuse-42.3_qemux86-64_20190126152612
There were two reasons for this:
a) the results of the selftest should be independent of which HOST_DISTRO
they're run on so they can be compared.
b) some builds only have one oe-selftest (a-quick) and some have four (a-full).
In an a-quick build, the HOST_DISTRO would likely therefore be different
between two builds but we still would like the tool to compare them.
> 2. For "resulttool store", I had noticed that it will now generally
> stored testresults.json in a meaningful file directory structure based
> on the store_map except oeselftest. oeselftest currently store
> multiple result id set inside oselftest file directory without
> comprehend the host distro.
>
> For example runtime, store testresult.json with the configured
> store_map.
> ├── oeselftest
> │ └── testresults.json
> ├── runtime
> │ ├── poky
> │ │ ├── qemuarm
> │ │ │ ├── core-image-minimal
> │ │ │ │ └── testresults.json
> │ │ │ ├── core-image-sato
> │ │ │ │ └── testresults.json
> │ │ │ └── core-image-sato-sdk
> │ │ │ └── testresults.json
> │ │ ├── qemuarm64
> │ │ │ ├── core-image-minimal
> │ │ │ │ └── testresults.json
> │ │ │ ├── core-image-sato
> │ │ │ │ └── testresults.json
> │ │ │ └── core-image-sato-sdk
> │ │ │ └── testresults.json
>
> I believe that we shall again comprehend the 'HOST_DISTRO'
> configuration inside the store_map.
> store_map = {
> -"oeselftest": ['TEST_TYPE'],
> +"oeselftest": ['TEST_TYPE','HOST_DISTRO'],
> "runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'],
> "sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
> "sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE',
> 'IMAGE_BASENAME']
>
> Doing so, it will store oeselftest in a more useful file directory
> structure with host distro comprehended.
> └── oeselftest
> ├── fedora-26
> │ └── testresults.json
> ├── fedora-28
> │ └── testresults.json
> ├── opensuse-42.3
> │ └── testresults.json
> └── ubuntu-18.04
> └── testresults.json
The reasoning is the same as the above, its more useful to allow the files to
be directly compared between different host distros.
Cheers,
Richard
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
On Mon, 2019-02-18 at 08:33 +, Yeoh, Ee Peng wrote:
> Hi RP,
>
> I have a question for "TESTSERIES".
> * Formalised the handling of "file_name" to "TESTSERIES" which the
> code will now add into the json configuration data if its not
> present, based on the directory name.
>
> May I know why was "TESTSERIES" was added as one of the key
> configuration for regression comparison selection inside
> regression_map?
> regression_map = {
> "oeselftest": ['TEST_TYPE', 'MACHINE'],
> "runtime": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME',
> 'MACHINE', 'IMAGE_PKGTYPE', 'DISTRO'],
> "sdk": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE',
> 'SDKMACHINE'],
> "sdkext": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME',
> 'MACHINE', 'SDKMACHINE']
> }
>
> Firstly, from the current yocto-testresults repository, I noticed
> that "TESTSERIES" was mostly duplicated with "MACHINE", or "MACHINE"
> & "DISTRO", or "TEST_TYPE" for selftest case.
The store_map doesn't use TESTSERIES as a key since I didn't think it
was needed for the git file layout. Particuarly for the runtime tests,
it does mean we end up with a larger number of results under the
qemux86* files in particular.
When performing regression analysis it seemed like a useful way to know
which results to compare and lowered the number of inexact matches we
had to make.
> Secondly, since "TESTSERIES" was created based on directory name from
> the source directory being used, will this introduce unexpected
> complication to regression comparison in the future if directory name
> for the source was changed? If directory name was changed even
> slightly, example for runtime_core-image-lsb, if the source directory
> name changed from "qemuarm-lsb" to "qemuarm_lsb", I believe the
> regression comparison will not able to compare the result id set even
> though they were having same configurations and they were meant to be
> compare directly.
For Yocto QA usage, the directory names are going to come directly from
the autobuilder target names so they should be consistent. I'd be fine
with adding a parameter to control it, right now it does add useful
information we need for the autobuilder though as it makes the results
more accurate (and hints to the user which autobuilder target the
change came from too).
Cheers,
Richard
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi Ee Peng,
On Mon, 2019-02-18 at 08:09 +, Yeoh, Ee Peng wrote:
> I did some testing with the latest from resulttool: Update to use
> gitarchive library function.
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=b9eecaabe56db5bcafff31e67cdabadc42e2d2e4
>
> I had 2 questions.
> 1. For "resulttool regression", currently it was comparing result id
> set without comprehending the difference in the host distro used to
> executed the oeselftest. Example: it was matching oeselftest run with
> fedora28 host distro with oeselftest run with ubuntu18 host distro,
> is this the expected behavior?
> Match: oeselftest_fedora-28_qemux86-64_20190201181656
>oeselftest_ubuntu-18.04_qemux86-64_20190201175023
> Match: oeselftest_fedora-26_qemux86-64_20190131144317
>oeselftest_fedora-26_qemux86-64_20190131144317
> Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
>oeselftest_fedora-28_qemux86-64_20190201181656
> Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
>oeselftest_opensuse-42.3_qemux86-64_20190126152612
There were two reasons for this:
a) the results of the selftest should be independent of which
HOST_DISTRO they're run on so they can be compared.
b) some builds only have one oe-selftest (a-quick) and some have four
(a-full). In an a-quick build, the HOST_DISTRO would likely therefore
be different between two builds but we still would like the tool to
compare them.
> 2. For "resulttool store", I had noticed that it will now generally
> stored testresults.json in a meaningful file directory structure
> based on the store_map except oeselftest. oeselftest currently store
> multiple result id set inside oselftest file directory without
> comprehend the host distro.
>
> For example runtime, store testresult.json with the configured
> store_map.
> ├── oeselftest
> │ └── testresults.json
> ├── runtime
> │ ├── poky
> │ │ ├── qemuarm
> │ │ │ ├── core-image-minimal
> │ │ │ │ └── testresults.json
> │ │ │ ├── core-image-sato
> │ │ │ │ └── testresults.json
> │ │ │ └── core-image-sato-sdk
> │ │ │ └── testresults.json
> │ │ ├── qemuarm64
> │ │ │ ├── core-image-minimal
> │ │ │ │ └── testresults.json
> │ │ │ ├── core-image-sato
> │ │ │ │ └── testresults.json
> │ │ │ └── core-image-sato-sdk
> │ │ │ └── testresults.json
>
> I believe that we shall again comprehend the 'HOST_DISTRO'
> configuration inside the store_map.
> store_map = {
> -"oeselftest": ['TEST_TYPE'],
> +"oeselftest": ['TEST_TYPE','HOST_DISTRO'],
> "runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'],
> "sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
> "sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE',
> 'IMAGE_BASENAME']
>
> Doing so, it will store oeselftest in a more useful file directory
> structure with host distro comprehended.
> └── oeselftest
> ├── fedora-26
> │ └── testresults.json
> ├── fedora-28
> │ └── testresults.json
> ├── opensuse-42.3
> │ └── testresults.json
> └── ubuntu-18.04
> └── testresults.json
The reasoning is the same as the above, its more useful to allow the
files to be directly compared between different host distros.
Cheers,
Richard
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
ot;master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-poky": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-selftest": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
},
"meta-yocto-bsp": {
"branch": "master",
"commit": "5fa3b5b15229babc9f96606c79436ab83651bf83",
"commit_count": 53265
}
},
"MACHINE": "qemux86-64",
"STARTTIME": "20190215010815",
"TESTSERIES": "oe-selftest",
"TEST_TYPE": "oeselftest"
},
Please let me know if you have any question for the above.
Best regards,
Yeoh Ee Peng
-Original Message-
From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org]
Sent: Monday, February 18, 2019 12:10 AM
To: Yeoh, Ee Peng ;
openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
On Thu, 2019-02-14 at 13:50 +0800, Yeoh Ee Peng wrote:
> v1:
> Face key error from oe-git-archive
> Undesirable behavior when storing to multiple git branch
>
> v2:
> Include fix for oe-git-archive
> Include fix for store result to multiple git branch
> Improve git commit message
>
> v3:
> Enhance fix for oe-git-archive by using exception catch to
> improve code readability and easy to understand
>
> v4:
> Add new features, merge result files & regression analysis
> Add selftest to merge, store, report and regression functionalities
> Revise codebase for pythonic
>
> v5:
> Add required files for selftest testing store
>
> v6:
> Add regression for directory and git repository
> Enable regression pairing base set to multiple target sets
> Revise selftest testing for regression
>
> v7:
> Optimize regression computation for ptest results
> Rename entry point script to resulttool
>
> Mazliana (1):
> scripts/resulttool: enable manual execution and result creation
>
> Yeoh Ee Peng (1):
> resulttool: enable merge, store, report and regression analysis
Hi Ee Peng,
Thanks for working on this, it does get better each iteration. I've been
struggling a little to explain what we need to do to finish this off. Firstly I
wanted to give some feedback on some general python
tips:
a) We can't use subprocess.run() as its a python 3.6 feature and we have
autobuilder workers with 3.5. This lead to failures like:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/242
We can use check_call or other functions instead.
b) I'd not recommend using "file" as a variable name in python as its a
keyword, similarly "dict" (in resultutils.py).
c) get_dict_value() is something we can probably avoid needing if we use the
.get() methods of dicts (you can specify a value to return if a value isn't
present).
I started to experiment with the tool to try and get it to follow the workflow
we need with the autobuilder QA process. Right now I'm heavily focusing on what
we need it to do to generate reports from the autobuilder, to the extent that
I'm ignoring most other workflows.
The reason for this is that I want to get it merged and use this to run
2.7 M3 testing on the autobuilder. The other workflows can be added if/as/when
we find we have need of them.
I ended up making a few changes to alter the tool to do the things I think we
need it to and to improve its output/usability. I'll send out a separate patch
with my changes so far. I've tried to summarise some of the reasoning here:
* Rename resultsutils -> resultutils to match the resultstool -> resulttool
rename
* Formalised the handling of "file_name" to "TESTSERIES" which the code will
now add into the json configuration data if its not present, based on the
directory name.
* When we don't have failed test cases, print something saying so instead of an
empty table
* Tweak the table headers in the report to be more readable (reference "Test
Series" instead if file_id and ID instead of results_id)
* Improve/simplify the max string length handling
* Merge the counts and percentage data into one table in the report since
pri
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi RP,
Thank you very much again for continuously providing your precious feedbacks to
me.
Also thank you very much for spending great amount of time to improve this
patchset siginificantly.
I did some testing with the latest from resulttool: Update to use gitarchive
library function.
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=b9eecaabe56db5bcafff31e67cdabadc42e2d2e4
I had 2 questions.
1. For "resulttool regression", currently it was comparing result id set
without comprehending the difference in the host distro used to executed the
oeselftest. Example: it was matching oeselftest run with fedora28 host distro
with oeselftest run with ubuntu18 host distro, is this the expected behavior?
Match: oeselftest_fedora-28_qemux86-64_20190201181656
oeselftest_ubuntu-18.04_qemux86-64_20190201175023
Match: oeselftest_fedora-26_qemux86-64_20190131144317
oeselftest_fedora-26_qemux86-64_20190131144317
Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
oeselftest_fedora-28_qemux86-64_20190201181656
Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
oeselftest_opensuse-42.3_qemux86-64_20190126152612
I believe that we shall comprehend the 'HOST_DISTRO' configuration inside the
regression_map.
regression_map = {
-"oeselftest": ['TEST_TYPE', 'MACHINE'],
+"oeselftest": ['TEST_TYPE', 'HOST_DISTRO', 'MACHINE'],
"runtime": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE',
'IMAGE_PKGTYPE', 'DISTRO'],
"sdk": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE',
'SDKMACHINE'],
"sdkext": ['TESTSERIES', 'TEST_TYPE', 'IMAGE_BASENAME', 'MACHINE',
'SDKMACHINE']
}
After comprehending this 'HOST_DISTRO', it was able to perform regression for
oeselftest with the matching host distro.
Match: oeselftest_ubuntu-18.04_qemux86-64_20190201175023
oeselftest_ubuntu-18.04_qemux86-64_20190201175023
Match: oeselftest_opensuse-42.3_qemux86-64_20190126152612
oeselftest_opensuse-42.3_qemux86-64_20190126152612
Match: oeselftest_fedora-26_qemux86-64_20190131144317
oeselftest_fedora-26_qemux86-64_20190131144317
Match: oeselftest_fedora-28_qemux86-64_20190201181656
oeselftest_fedora-28_qemux86-64_20190201181656
2. For "resulttool store", I had noticed that it will now generally stored
testresults.json in a meaningful file directory structure based on the
store_map except oeselftest. oeselftest currently store multiple result id set
inside oselftest file directory without comprehend the host distro.
For example runtime, store testresult.json with the configured store_map.
├── oeselftest
│ └── testresults.json
├── runtime
│ ├── poky
│ │ ├── qemuarm
│ │ │ ├── core-image-minimal
│ │ │ │ └── testresults.json
│ │ │ ├── core-image-sato
│ │ │ │ └── testresults.json
│ │ │ └── core-image-sato-sdk
│ │ │ └── testresults.json
│ │ ├── qemuarm64
│ │ │ ├── core-image-minimal
│ │ │ │ └── testresults.json
│ │ │ ├── core-image-sato
│ │ │ │ └── testresults.json
│ │ │ └── core-image-sato-sdk
│ │ │ └── testresults.json
I believe that we shall again comprehend the 'HOST_DISTRO' configuration inside
the store_map.
store_map = {
-"oeselftest": ['TEST_TYPE'],
+"oeselftest": ['TEST_TYPE','HOST_DISTRO'],
"runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'],
"sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
"sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME']
Doing so, it will store oeselftest in a more useful file directory structure
with host distro comprehended.
└── oeselftest
├── fedora-26
│ └── testresults.json
├── fedora-28
│ └── testresults.json
├── opensuse-42.3
│ └── testresults.json
└── ubuntu-18.04
└── testresults.json
Please let me know if you have any question related to above.
Best regards,
Yeoh Ee Peng
-Original Message-
From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org]
Sent: Monday, February 18, 2019 6:46 AM
To: Yeoh, Ee Peng ;
openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
On Sun, 2019-02-17 at 17:54 +, Richard Purdie wrote:
> > Despite my changes there are things that still need to be done.
> > Essential things which need to happen before this code merges:
> >
> > * oe-git-archive is
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
Hi RP, Thank you very much for providing me your precious advices and I will definitely look into them. Let me look into all the improvements that you had developed and I will try my best to provide further improvement needed. Best regards, Yeoh Ee Peng -Original Message- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Monday, February 18, 2019 12:10 AM To: Yeoh, Ee Peng ; openembedded-core@lists.openembedded.org Subject: Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt On Thu, 2019-02-14 at 13:50 +0800, Yeoh Ee Peng wrote: > v1: > Face key error from oe-git-archive > Undesirable behavior when storing to multiple git branch > > v2: > Include fix for oe-git-archive > Include fix for store result to multiple git branch > Improve git commit message > > v3: > Enhance fix for oe-git-archive by using exception catch to > improve code readability and easy to understand > > v4: > Add new features, merge result files & regression analysis > Add selftest to merge, store, report and regression functionalities > Revise codebase for pythonic > > v5: > Add required files for selftest testing store > > v6: > Add regression for directory and git repository > Enable regression pairing base set to multiple target sets > Revise selftest testing for regression > > v7: > Optimize regression computation for ptest results > Rename entry point script to resulttool > > Mazliana (1): > scripts/resulttool: enable manual execution and result creation > > Yeoh Ee Peng (1): > resulttool: enable merge, store, report and regression analysis Hi Ee Peng, Thanks for working on this, it does get better each iteration. I've been struggling a little to explain what we need to do to finish this off. Firstly I wanted to give some feedback on some general python tips: a) We can't use subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. This lead to failures like: https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/242 We can use check_call or other functions instead. b) I'd not recommend using "file" as a variable name in python as its a keyword, similarly "dict" (in resultutils.py). c) get_dict_value() is something we can probably avoid needing if we use the .get() methods of dicts (you can specify a value to return if a value isn't present). I started to experiment with the tool to try and get it to follow the workflow we need with the autobuilder QA process. Right now I'm heavily focusing on what we need it to do to generate reports from the autobuilder, to the extent that I'm ignoring most other workflows. The reason for this is that I want to get it merged and use this to run 2.7 M3 testing on the autobuilder. The other workflows can be added if/as/when we find we have need of them. I ended up making a few changes to alter the tool to do the things I think we need it to and to improve its output/usability. I'll send out a separate patch with my changes so far. I've tried to summarise some of the reasoning here: * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropria
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
On Sun, 2019-02-17 at 17:54 +, Richard Purdie wrote: > > Despite my changes there are things that still need to be done. > > Essential things which need to happen before this code merges: > > > > * oe-git-archive is importing using the commit/branch of the > > current > > repo, not the data in the results file. Also now fixed. I put my patches into master-next too. With this working, I was able to run something along the lines of: for D in $1/*; do resulttool store $D $2 --allow-empty done on the autobuilder's recent results which lead to the creation of this repository: http://git.yoctoproject.org/cgit.cgi/yocto-testresults/ > > * Revisit and redo the way the git branch handling is happening. > > We > > really want to model how oe-build-perf-report handles git repos > > for > > comparisons: > > - Its able to query data from git repos without changing the > > current > > working branch, > > - it can search on tag formats to find comparison data Which means we now need to make the git branch functionality of the report and regression commands compare with the above repo, so we're a step closer to getting thie merged. Ultimately we'll auto-populate the above repo by having the autobuilder run a "store" command at the end of its runs. I have a feeling I may have broken the resulttool selftests so that is something else which will need to be fixed before anything merges. Time for me to step away from the keyboard for a bit too. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
> Despite my changes there are things that still need to be done. > Essential things which need to happen before this code merges: > > * oe-git-archive is importing using the commit/branch of the current > repo, not the data in the results file. > > * Fix the -t option to merge command Got rid of this for now, we can add it later if we need it, can become a "nice to have" for later. > * Audit the command option help Done on my branch. > * Revisit and redo the way the git branch handling is happening. We > really want to model how oe-build-perf-report handles git repos > for > comparisons: > - Its able to query data from git repos without changing the > current > working branch, > - it can search on tag formats to find comparison data > > * Add ptest summary to the report command Done on my branch. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 0/2 v7] test-case-mgmt
On Thu, 2019-02-14 at 13:50 +0800, Yeoh Ee Peng wrote: > v1: > Face key error from oe-git-archive > Undesirable behavior when storing to multiple git branch > > v2: > Include fix for oe-git-archive > Include fix for store result to multiple git branch > Improve git commit message > > v3: > Enhance fix for oe-git-archive by using exception catch to > improve code readability and easy to understand > > v4: > Add new features, merge result files & regression analysis > Add selftest to merge, store, report and regression functionalities > Revise codebase for pythonic > > v5: > Add required files for selftest testing store > > v6: > Add regression for directory and git repository > Enable regression pairing base set to multiple target sets > Revise selftest testing for regression > > v7: > Optimize regression computation for ptest results > Rename entry point script to resulttool > > Mazliana (1): > scripts/resulttool: enable manual execution and result creation > > Yeoh Ee Peng (1): > resulttool: enable merge, store, report and regression analysis Hi Ee Peng, Thanks for working on this, it does get better each iteration. I've been struggling a little to explain what we need to do to finish this off. Firstly I wanted to give some feedback on some general python tips: a) We can't use subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. This lead to failures like: https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/242 We can use check_call or other functions instead. b) I'd not recommend using "file" as a variable name in python as its a keyword, similarly "dict" (in resultutils.py). c) get_dict_value() is something we can probably avoid needing if we use the .get() methods of dicts (you can specify a value to return if a value isn't present). I started to experiment with the tool to try and get it to follow the workflow we need with the autobuilder QA process. Right now I'm heavily focusing on what we need it to do to generate reports from the autobuilder, to the extent that I'm ignoring most other workflows. The reason for this is that I want to get it merged and use this to run 2.7 M3 testing on the autobuilder. The other workflows can be added if/as/when we find we have need of them. I ended up making a few changes to alter the tool to do the things I think we need it to and to improve its output/usability. I'll send out a separate patch with my changes so far. I've tried to summarise some of the reasoning here: * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults) Despite my changes there are things that still need to be done. Essential things which need to happen before this code merges: * oe-git-archive is importing using the commit/branch of the current repo, not the data in the results file. * Fix the -t option to merge command * Audit the command option help * Revisit and redo the way the git branch handling is happening. We really want to model how oe-build-perf-report handles git repos for comparisons: - Its able to query data from git repos without changing the current working branch, - it can search on tag formats to find comparison data * Add ptest summary to the report command Things which may be "nice to have" which can come i
