[OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.
These scripts were developed as a test result tools to manage
these testresults.json file.
Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json
files to a target file.
Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.
Using the "regression" operation, user can perform regression analysis
on testresults.json files specified.
These resultstool operations expect the testresults.json file to use
the json format below.
{
"": {
"configuration": {
"": "",
"": "",
...
"": "",
},
"result": {
"": {
"status": "",
"log": ""
},
"": {
"status": "",
"log": ""
},
...
"": {
"status": "",
"log": ""
},
}
},
...
"": {
"configuration": {
"": "",
"": "",
...
"": "",
},
"result": {
"": {
"status": "",
"log": ""
},
"": {
"status": "",
"log": ""
},
...
"": {
"status": "",
"log": ""
},
}
},
}
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ resultstool
To store test result from oeqa automated tests, execute the below
$ resultstool store
To merge multiple testresults.json files, execute the below
$ resultstool merge
To report test report, execute the below
$ resultstool report
To perform regression analysis, execute the below
$ resultstool regression
[YOCTO# 13012]
[YOCTO# 12654]
Signed-off-by: Yeoh Ee Peng
---
meta/lib/oeqa/files/testresults/testresults.json | 40 ++
meta/lib/oeqa/selftest/cases/resultstooltests.py | 104
scripts/lib/resultstool/__init__.py| 0
scripts/lib/resultstool/merge.py | 71 +++
scripts/lib/resultstool/regression.py | 134 +
scripts/lib/resultstool/report.py | 122 +++
scripts/lib/resultstool/resultsutils.py| 47
scripts/lib/resultstool/store.py | 110 +
.../resultstool/template/test_report_full_text.txt | 35 ++
scripts/resultstool| 84 +
10 files changed, 747 insertions(+)
create mode 100644 meta/lib/oeqa/files/testresults/testresults.json
create mode 100644 meta/lib/oeqa/selftest/cases/resultstooltests.py
create mode 100644 scripts/lib/resultstool/__init__.py
create mode 100644 scripts/lib/resultstool/merge.py
create mode 100644 scripts/lib/resultstool/regression.py
create mode 100644 scripts/lib/resultstool/report.py
create mode 100644 scripts/lib/resultstool/resultsutils.py
create mode 100644 scripts/lib/resultstool/store.py
create mode 100644 scripts/lib/resultstool/template/test_report_full_text.txt
create mode 100755 scripts/resultstool
diff --git a/meta/lib/oeqa/files/testresults/testresults.json
b/meta/lib/oeqa/files/testresults/testresults.json
new file mode 100644
index 000..1a62155
--- /dev/null
+++ b/meta/lib/oeqa/files/testresults/testresults.json
@@ -0,0 +1,40 @@
+{
+"runtime_core-image-minimal_qemuarm_20181225195701": {
+"configuration": {
+"DISTRO": "poky",
+"HOST_DISTRO": "ubuntu-16.04",
+"IMAGE_BASENAME": "core-image-minimal",
+"IMAGE_PKGTYPE": "rpm",
+"LAYERS": {
+"meta": {
+"branch": "master",
+"commit": "801745d918e83f976c706f29669779f5b292ade3",
+"commit_count": 52782
+},
+"meta-poky": {
+"branch": "master",
+"commit": "801745d918e83f976c706f29669779f5b292ade3",
+"commit_count": 52782
+},
+"meta-yocto-bsp": {
+"branch": "master",
+"commit": "801745d918e83f976c706f29669779f5b292ade3",
+"commit_count": 52782
+}
+},
+"MACHINE": "qemuarm",
+"STARTTIME": "20181225195701",
+"TEST_TYPE": "runtime"
+},
+"result": {
+"apt.AptRe
Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
RP,
The current patch allow files based regression, meaning if you have file#1 and
file#2, the regression will select result instances for regression based on the
configuration data available.
There are 2 more regression use cases that I have in mind:
Use case#1: directory based regression - Assumed that there are 2 directories
storing list of result files. User shall provide these 2 directories for
regression, regression scripts will first parse through all the available files
inside each directories, then perform regression based on available
configuration data to determine the regression pair (eg. select result_set_1
from directory#1 and result_set_x from directory#2 if they both have matching
configurations).
Use case#2: git branch based regression - Assumed that result files are stored
inside git repository with specific git branch storing result files for single
commit. User shall provide the 2 specific git branches for regression,
regression scripts will first parse through all the available files inside each
git branch, then perform regression based on available configuration data to
determine the regression pair (eg. select result_set_1 from git_branch_1 and
result_set_x from git_branch_2 if they both have matching configurations).
Any idea which regression use cases above was needed? We shall develop the next
regression functionalities based on your inputs.
Best regards,
Yeoh Ee Peng
-Original Message-
From: Yeoh, Ee Peng
Sent: Tuesday, January 22, 2019 5:42 PM
To: openembedded-core@lists.openembedded.org
Cc: Yeoh, Ee Peng
Subject: [PATCH 1/2 v5] resultstool: enable merge, store, report and regression
analysis
OEQA outputs test results into json files and these files were archived by
Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder
for different host distro generate a testresults.json file.
These scripts were developed as a test result tools to manage these
testresults.json file.
Using the "store" operation, user can store multiple testresults.json files as
well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json files to
a target file.
Using the "report" operation, user can view the test result summary for all
available testresults.json files inside a ordinary directory or a git
repository.
Using the "regression" operation, user can perform regression analysis on
testresults.json files specified.
These resultstool operations expect the testresults.json file to use the json
format below.
{
"": {
"configuration": {
"": "",
"": "",
...
"": "",
},
"result": {
"": {
"status": "",
"log": ""
},
"": {
"status": "",
"log": ""
},
...
"": {
"status": "",
"log": ""
},
}
},
...
"": {
"configuration": {
"": "",
"": "",
...
"": "",
},
"result": {
"": {
"status": "",
"log": ""
},
"": {
"status": "",
"log": ""
},
...
"": {
"status": "",
"log": ""
},
}
},
}
To use these scripts, first source oe environment, then run the entry point
script to look for help.
$ resultstool
To store test result from oeqa automated tests, execute the below
$ resultstool store
To merge multiple testresults.json files, execute the below
$ resultstool merge
To report test report, execute the below
$ resultstool report
To perform regression analysis, execute the below
$ resultstool regression
[YOCTO# 13012]
[YOCTO# 12654]
Signed-off-by: Yeoh Ee Peng
---
meta/lib/oeqa/files/testresults/testresults.json | 40 ++
meta/lib/oeqa/selftest/cases/resultstooltests.py | 104
scripts/lib/resultstool/__init__.py| 0
scripts/lib/resultstool/merge.py | 71 +++
scripts/lib/resultstool/regression.py | 134 +
scripts/lib/resultstool/report.py | 122 +++
scripts/lib/resultstool/resultsutils.py| 47
scripts/lib/resultstool/store.py | 110 +
.../resultstool/template/test_report_full_text.txt | 35 ++
scripts/resultstool| 84 +
10 files changed, 747 insertions(+)
create mode 100644 meta/lib/oeqa/files/testresults/testresults.json
create mode 100644 meta/lib/oeqa/selftest/cases/resultstooltests.py
create mode 100644 scripts/lib/resultstool/__init__.py
create mode 100644 scripts/lib/resultstool/mer
Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
On Tue, 2019-01-22 at 17:42 +0800, Yeoh Ee Peng wrote: > OEQA outputs test results into json files and these files were > archived by Autobuilder during QA releases. Example: each oe-selftest > run by Autobuilder for different host distro generate a > testresults.json file. > > These scripts were developed as a test result tools to manage > these testresults.json file. > > Using the "store" operation, user can store multiple testresults.json > files as well as the pre-configured directories used to hold those > files. > > Using the "merge" operation, user can merge multiple testresults.json > files to a target file. > > Using the "report" operation, user can view the test result summary > for all available testresults.json files inside a ordinary directory > or a git repository. > > Using the "regression" operation, user can perform regression > analysis > on testresults.json files specified. Thanks Ee Peng, this version is much improved! As an experiment I had a local test results file and I was able to run: $ resultstool regression /tmp/repo/testresults.json /tmp/repo/testresults.json -b sdk_core-image-sato_x86_64_qemumips_20181219111311 -t sdk_core-image-sato_x86_64_qemumips_20181219200052 Successfully loaded base test results from: /tmp/repo/testresults.json Successfully loaded target test results from: /tmp/repo/testresults.json Getting base test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219111311 Getting target test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219200052 Start Regression Only print regression if base status not equal target : -> assimp.BuildAssimp.test_assimp : ERROR -> PASSED ==End Regression== I was able to clearly see that my failing test case went from ERROR to PASSED which is good. The interface and the way the information is being presented and stored are now the things we need to work on. What is odd about the current tool/behaviour is that it sometimes expects files that are stored in a repository and sometimes expects standalone files. It doesn't feel like the work flow and the way the user would interact with this is quite correct. You can see from my test case above that the results file I wanted to compare were both in the same repo file which I'd already merged/stored. The tool can do it by passing the file twice and specifying the IDs but its awkward and not obvious. I'm still a little unsure how we expect to use the tool and whether the layout of the files in the git repository is how we're going to ultimately want to do this. For example, combining all results into a single json file means we can't really show useful comparison information directly from "git diff". Would we be better with a file per results ID for example with a layout allowing "git diff" to be useful? A directory structure like: selftest// runtime// sdk// esdk// build-perf// for example? I don't have the "right" answer here as I've not experimented with this but its something we need to think about. As another example, the regression command can't compare the file from two different git commits easily, you have to copy one file out the git repo and then checkout the other version to compare them. This isn't easy for the user. In summary, I think the base underlying tools here are heading the right way but the layout of the git repo and the way people are expected to use and interact with it needs a little bit more work... Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
As a further example of this I wanted to experiment with a real world example. We have: https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86/testresults.json https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86/testresults.json which are all results files for the same thing from 2.6.1, 2.7M1 and 2.7M2. I also have one for 2.6.0 locally but its the same as 2.6.1. Running this gives: resultstool regression qemux86-2.6.json qemux86-2.7-M2.json Successfully loaded base test results from: qemux86-2.6.json Successfully loaded target test results from: qemux86-2.7-M2.json Getting regression for base=runtime_core-image-minimal_qemux86_20181029181546 target=runtime_core-image-minimal_qemux86_20190124185230 Start Regression Only print regression if base status not equal target : -> ==End Regression== Getting regression for base=runtime_core-image-sato-sdk_qemux86_20181029181546 target=runtime_core-image-sato-sdk_qemux86_20190124185230 Start Regression Only print regression if base status not equal target : -> ==End Regression== Getting regression for base=runtime_core-image-sato_qemux86_20181029181546 target=runtime_core-image-sato_qemux86_20190124185230 Start Regression Only print regression if base status not equal target : -> ==End Regression== Getting regression for base=sdk_core-image-sato_i686_qemux86_20181029190813 target=sdk_core-image-sato_i686_qemux86_20190124185230 Start Regression Only print regression if base status not equal target : -> python.PythonTest.test_python3 : UNKNOWN -> None ==End Regression== Getting regression for base=sdk_core-image-sato_x86_64_qemux86_20181029192305 target=sdk_core-image-sato_x86_64_qemux86_20190124192316 Start Regression Only print regression if base status not equal target : -> python.PythonTest.test_python3 : UNKNOWN -> None ==End Regression== Getting regression for base=sdkext_core-image-minimal_x86_64_qemux86_20181029192305 target=sdkext_core-image-minimal_x86_64_qemux86_20190124192316 Start Regression Only print regression if base status not equal target : -> python.PythonTest.test_python3 : PASSED -> None ==End Regression== Getting regression for base=sdkext_core-image-sato_x86_64_qemux86_20181029192305 target=sdkext_core-image-sato_x86_64_qemux86_20190124192316 Start Regression Only print regression if base status not equal target : -> python.PythonTest.test_python3 : PASSED -> None ==End Regression== which shows that: a) In the case of no regressions it may be better simply print that rather than a header and empty body b) It does highlight that the "python3" test somehow disappeared c) Its hard to know that in fact another python3 test started passing. Maybe we also need a compare command to show "new" results to make it easier to investigate these things? For interest I ran the comparision backwards: resultstool regression qemux86-2.7-M2.json qemux86-2.6.json Successfully loaded base test results from: qemux86-2.7-M2.json Successfully loaded target test results from: qemux86-2.6.json Getting regression for base=runtime_core-image-minimal_qemux86_20190124185230 target=runtime_core-image-minimal_qemux86_20181029181546 Start Regression Only print regression if base status not equal target : -> apt.AptRepoTest.test_apt_install_from_repo : SKIPPED -> None buildcpio.BuildCpioTest.test_cpio : SKIPPED -> None buildgalculator.GalculatorTest.test_galculator : SKIPPED -> None buildlzip
Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi RP, Thanks for providing the precious inputs. Agreed with you that the current patch that enable files based regression was not enough for other use cases. >From the information that you had shared, there are 2 more regression use >cases that I have in mind: Use case#1: directory based regression Given that Autobuilder stored result files inside /testresults directories, user shall be able to perform the directory based regression using output from Autobuilder directly, such as below Autobuilder directories. https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86/testresults.json https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86/testresults.json Assumed that there are 2 directories storing list of result files. User shall provide these 2 directories for regression, regression scripts will first parse through all the available files inside each directories, then perform regression based on available configuration data to determine the regression pair (eg. select result_set_1 from directory#1 and result_set_x from directory#2 if they both have matching configurations). Use case#2: git branch based regression Given that Autobuilder stored result files inside /testresults directories, user shall first store these directories and the result files in each git branch accordingly using the existing store plugin. After that, user can used the git branch based regression to analysis the information. Store in yocto-2.6.1.rc1, yocto-2.7_M1.rc1, yocto-2.7_M2.rc1 git branch accordingly https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/ https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/ https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/ Assumed that result files are stored inside git repository with specific git branch storing result files for single commit. User shall provide the 2 specific git branches for regression, regression scripts will first parse through all the available files inside each git branch, then perform regression based on available configuration data to determine the regression pair (eg. select result_set_1 from git_branch_1 and result_set_x from git_branch_2 if they both have matching configurations). The current codebase can be easily extended to enable both use cases above. Please let me know if both use cases above are important and please give us your inputs. Thanks, Ee Peng -Original Message- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Friday, January 25, 2019 11:44 PM To: Yeoh, Ee Peng ; openembedded-core@lists.openembedded.org Cc: Eggleton, Paul ; Burton, Ross Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis On Tue, 2019-01-22 at 17:42 +0800, Yeoh Ee Peng wrote: > OEQA outputs test results into json files and these files were > archived by Autobuilder during QA releases. Example: each oe-selftest > run by Autobuilder for different host distro generate a > testresults.json file. > > These scripts were developed as a test result tools to manage these > testresults.json file. > > Using the "store" operation, user can store multiple testresults.json > files as well as the pre-configured directories used to hold those > files. > > Using the "merge" operation, user can merge multiple testresults.json > files to a target file. > > Using the "report" operation, user can view the test result summary > for all available testresults.json files inside a ordinary directory > or a git repository. > > Using the "regression" operation, user can perform regression analysis > on testresults.json files specified. Thanks Ee Peng, this version is much improved! As an experiment I had a local test results file and I was able to run: $ resultstool regression /tmp/repo/testresults.json /tmp/repo/testresults.json -b sdk_core-image-sato_x86_64_qemumips_20181219111311 -t sdk_core-image-sato_x86_64_qemumips_20181219200052 Successfully loaded base test results from: /tmp/repo/testresults.json Successfully loaded target test results from: /tmp/repo/testresults.json Getting base test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219111311 Getting target test result with result_id=sdk_core-image-sato_x86_64_qemumips_20181219200052 Start Regression Only print regression if base status not equal target : -> assimp.BuildAssimp.test_assimp : ERROR -> PASSED ==End Regression== I was able to clearly see that my failing test case went from ERROR to PA
Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi Ee Peng, On Mon, 2019-01-28 at 02:12 +, Yeoh, Ee Peng wrote: > Thanks for providing the precious inputs. > Agreed with you that the current patch that enable files based > regression was not enough for other use cases. > > From the information that you had shared, there are 2 more regression > use cases that I have in mind: > Use case#1: directory based regression > Given that Autobuilder stored result files inside /testresults > directories, user shall be able to perform the directory based > regression using output from Autobuilder directly, such as below > Autobuilder directories. > https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86/testresults.json > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86/testresults.json > > Assumed that there are 2 directories storing list of result files. > User shall provide these 2 directories for regression, regression > scripts will first parse through all the available files inside each > directories, then perform regression based on available configuration > data to determine the regression pair (eg. select result_set_1 from > directory#1 and result_set_x from directory#2 if they both have > matching configurations). Yes, this would be very useful. I suspect you don't need to have matching layouts, just import from all the json files in a given directory for the comparison. This way we can support arbitrary layouts. > Use case#2: git branch based regression > Given that Autobuilder stored result files inside /testresults > directories, user shall first store these directories and the result > files in each git branch accordingly using the existing store plugin. > After that, user can used the git branch based regression to analysis > the information. > Store in yocto-2.6.1.rc1, yocto-2.7_M1.rc1, yocto-2.7_M2.rc1 git > branch accordingly > https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/ > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/ > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/ > > Assumed that result files are stored inside git repository with > specific git branch storing result files for single commit. User > shall provide the 2 specific git branches for regression, regression > scripts will first parse through all the available files inside each > git branch, then perform regression based on available configuration > data to determine the regression pair (eg. select result_set_1 from > git_branch_1 and result_set_x from git_branch_2 if they both have > matching configurations). > > The current codebase can be easily extended to enable both use cases > above. Please let me know if both use cases above are important and > please give us your inputs. Yes, I think these are two key use cases we need to support. I think there is a third thing we also need to look at: It would be great if there was some way of allowing some kind of templating when storing into the git repository. This way a general local log file from tmp/log/oeqa could be stored into the git repo, being split according to the layout of the repo if needed. Our default layout could match that from the autobuilder but the repository could define a layout? As mentioned, we also need to think about ptest. Currently the runtime ptest code has some parsing and log generation code. I think pieces like: http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101 and the log_as_files() code should be moved to the reporting tool and that the test code should just generate the json results file which can later be parsed/processed as needed. I did post on the list earlier about some of the other challenges we have with the ptest data. buildperf is the other piece of this which we'll need to think about. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi RP, I had submitted the v6 patches with below changes. v6: Add regression for directory and git repository Enable regression pairing base set to multiple target sets Revise selftest testing for regression http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278486.html http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278487.html http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278488.html For regression directory and git, it can support arbitrary directory layout. The regression will select pair of result instances for comparison based on the unique configurations data inside the result instance itself. I have some questions regarding below items: >I think there is a third thing we also need to look at: > >It would be great if there was some way of allowing some kind of templating >when storing into the git >repository. This way a general local log file from >tmp/log/oeqa could be stored into the git repo, being >split according to the >layout of the repo if needed. > >Our default layout could match that from the autobuilder but the repository >could define a layout? Before developing custom template layout for store git repo, I would like to understand more so that I will make sure the output will fulfill the requirement. May I know what was the intention to store result into git repo with custom layout template? Can you share the use case? For ptest and perform tests, let me look into them. Thank you for sharing the logparser. http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101 Best regards, Yeoh Ee Peng -Original Message- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Tuesday, January 29, 2019 12:29 AM To: Yeoh, Ee Peng ; openembedded-core@lists.openembedded.org Cc: Eggleton, Paul ; Burton, Ross Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis Hi Ee Peng, On Mon, 2019-01-28 at 02:12 +, Yeoh, Ee Peng wrote: > Thanks for providing the precious inputs. > Agreed with you that the current patch that enable files based > regression was not enough for other use cases. > > From the information that you had shared, there are 2 more regression > use cases that I have in mind: > Use case#1: directory based regression Given that Autobuilder stored > result files inside /testresults directories, user shall be able to > perform the directory based regression using output from Autobuilder > directly, such as below Autobuilder directories. > https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/ > qemux86/testresults.json > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults > /qemux86/testresults.json > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults > /qemux86/testresults.json > > Assumed that there are 2 directories storing list of result files. > User shall provide these 2 directories for regression, regression > scripts will first parse through all the available files inside each > directories, then perform regression based on available configuration > data to determine the regression pair (eg. select result_set_1 from > directory#1 and result_set_x from directory#2 if they both have > matching configurations). Yes, this would be very useful. I suspect you don't need to have matching layouts, just import from all the json files in a given directory for the comparison. This way we can support arbitrary layouts. > Use case#2: git branch based regression Given that Autobuilder stored > result files inside /testresults directories, user shall first store > these directories and the result files in each git branch accordingly > using the existing store plugin. > After that, user can used the git branch based regression to analysis > the information. > Store in yocto-2.6.1.rc1, yocto-2.7_M1.rc1, yocto-2.7_M2.rc1 git > branch accordingly > https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/ > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults > / > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults > / > > Assumed that result files are stored inside git repository with > specific git branch storing result files for single commit. User shall > provide the 2 specific git branches for regression, regression scripts > will first parse through all the available files inside each git > branch, then perform regression based on available configuration data > to determine the regression pair (eg. select result_set_1 from > git_branch_1 and result_set_x from git_branch_2 if they both have > matching configurations). > > The current codebase can be easily extended to enable
Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
On Thu, 2019-01-31 at 05:23 +, Yeoh, Ee Peng wrote: > Hi RP, > > I looked into ptest and regression. The existing "resultstool > regression" can be used to perform regression on ptest, since the > testresults.json capture ptest status. I had executed regression > script for the below 2 ptest testresults.json. Attached was the > regression report for ptest. > > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86-64-ptest/testresults.json > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86-64-ptest/testresults.json > > The only challenges now was since ptest result set was relatively > large, it was taking some time for computing the regression. Also > there was this "ptestresult.rawlogs" testcase that does not contain > status but the large rawlog. > > I did an experiment where I run the regression on testresults.json > with and without the ptest rawlog. It shows the time taken for > regression was significantly larger when it contain the rawlog. I > will try to improve the regression time by throw away the rawlog at > runtime when perform computing. > testresults.json with rawlog > Regression start time: 20190131122805 > Regression end time: 20190131124425 > Time taken: 16 mins 20 sec > > testresults.json without rawlog > Regression start time: 20190131124512 > Regression end time: 20190131124529 > Time taken: 17 sec Analysing the rawlog makes no sense so the tool needs to simply ignore that. 16 minutes is far too long! I've just merged some changes which mean there are probably some other sections it will need to ignore now too since the logs are now being split out per ptest (section). I've left rawlogs in as its useful for debugging but once the section splits are working we could remove it. This adds in timing data so we know how long each ptest took to run (in seconds), it also adds in exit code and timeout data. These all complicate the regression analysis but the fact that lttng has been timing out (for example) has been overlooked until now and shows we need to analyse these things. I'm considering whether we should have a command in resulttool which takes json data and writes it out in a "filesystem" form. The code in logparser.py already has a rudimentary version of this for ptest data. It could be extended to write out a X.log for each ptest based on the split out data and maybe duration and timeout information in some form too. The idea behind flat filesystem representations of the data is that a user can more easily explore or compare them, they also show up well in git. Its also worth thinking about how we'll end up using this. testresult will get called at the end of builds (particularly) release builds and we'll want it to generate a QA report for the automated test data. The autobuilder will likely put an http link in the "release build ready" email to an html like report stored alongside the testresults json files. I'm still trying to figure out how to make this all fit together and allow automated comparisons but the build performance data would also fit into this (and already has html reports). Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis
Hi RP, I had executed the runtime/ptest using the latest master with the latest changes to understand the new improvement. For now, the resulttool regression will ignore both 'ptestresult.rawlogs' & 'ptestresult.sections' as current regression operation focuses on comparing the "status" differences and it does not need the log as well as the new section information. By ignoring both 'ptestresult.rawlogs' & 'ptestresult.sections' , the regression time was optimized to seconds instead of minutes for ptest. For additional information inside 'ptestresult.sections', do we need similar regression? Any idea which data inside 'ptestresult.sections' will be useful for regression? Currently, resulttool regression will only print text based report, if html report was needed, it can be extend by using jinja2 framework. Do we need html report for this regression? Any requirement for the html report? http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278971.html http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278972.html http://lists.openembedded.org/pipermail/openembedded-core/2019-February/278973.html Thanks, Yeoh Ee Peng -Original Message- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Friday, February 1, 2019 7:40 AM To: Yeoh, Ee Peng ; 'openembedded-core@lists.openembedded.org' Cc: Eggleton, Paul ; Burton, Ross Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis On Thu, 2019-01-31 at 05:23 +, Yeoh, Ee Peng wrote: > Hi RP, > > I looked into ptest and regression. The existing "resultstool > regression" can be used to perform regression on ptest, since the > testresults.json capture ptest status. I had executed regression > script for the below 2 ptest testresults.json. Attached was the > regression report for ptest. > > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults > /qemux86-64-ptest/testresults.json > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults > /qemux86-64-ptest/testresults.json > > The only challenges now was since ptest result set was relatively > large, it was taking some time for computing the regression. Also > there was this "ptestresult.rawlogs" testcase that does not contain > status but the large rawlog. > > I did an experiment where I run the regression on testresults.json > with and without the ptest rawlog. It shows the time taken for > regression was significantly larger when it contain the rawlog. I will > try to improve the regression time by throw away the rawlog at runtime > when perform computing. > testresults.json with rawlog > Regression start time: 20190131122805 > Regression end time: 20190131124425 > Time taken: 16 mins 20 sec > > testresults.json without rawlog > Regression start time: 20190131124512 > Regression end time: 20190131124529 > Time taken: 17 sec Analysing the rawlog makes no sense so the tool needs to simply ignore that. 16 minutes is far too long! I've just merged some changes which mean there are probably some other sections it will need to ignore now too since the logs are now being split out per ptest (section). I've left rawlogs in as its useful for debugging but once the section splits are working we could remove it. This adds in timing data so we know how long each ptest took to run (in seconds), it also adds in exit code and timeout data. These all complicate the regression analysis but the fact that lttng has been timing out (for example) has been overlooked until now and shows we need to analyse these things. I'm considering whether we should have a command in resulttool which takes json data and writes it out in a "filesystem" form. The code in logparser.py already has a rudimentary version of this for ptest data. It could be extended to write out a X.log for each ptest based on the split out data and maybe duration and timeout information in some form too. The idea behind flat filesystem representations of the data is that a user can more easily explore or compare them, they also show up well in git. Its also worth thinking about how we'll end up using this. testresult will get called at the end of builds (particularly) release builds and we'll want it to generate a QA report for the automated test data. The autobuilder will likely put an http link in the "release build ready" email to an html like report stored alongside the testresults json files. I'm still trying to figure out how to make this all fit together and allow automated comparisons but the build performance data would also fit into this (and already has html reports). Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
