Hi RP, I had submitted the v6 patches with below changes. v6: Add regression for directory and git repository Enable regression pairing base set to multiple target sets Revise selftest testing for regression http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278486.html http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278487.html http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278488.html
For regression directory and git, it can support arbitrary directory layout. The regression will select pair of result instances for comparison based on the unique configurations data inside the result instance itself. I have some questions regarding below items: >I think there is a third thing we also need to look at: > >It would be great if there was some way of allowing some kind of templating >when storing into the git >repository. This way a general local log file from >tmp/log/oeqa could be stored into the git repo, being >split according to the >layout of the repo if needed. > >Our default layout could match that from the autobuilder but the repository >could define a layout? Before developing custom template layout for store git repo, I would like to understand more so that I will make sure the output will fulfill the requirement. May I know what was the intention to store result into git repo with custom layout template? Can you share the use case? For ptest and perform tests, let me look into them. Thank you for sharing the logparser. http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101 Best regards, Yeoh Ee Peng -----Original Message----- From: Richard Purdie [mailto:richard.pur...@linuxfoundation.org] Sent: Tuesday, January 29, 2019 12:29 AM To: Yeoh, Ee Peng <ee.peng.y...@intel.com>; openembedded-core@lists.openembedded.org Cc: Eggleton, Paul <paul.eggle...@intel.com>; Burton, Ross <ross.bur...@intel.com> Subject: Re: [OE-core] [PATCH 1/2 v5] resultstool: enable merge, store, report and regression analysis Hi Ee Peng, On Mon, 2019-01-28 at 02:12 +0000, Yeoh, Ee Peng wrote: > Thanks for providing the precious inputs. > Agreed with you that the current patch that enable files based > regression was not enough for other use cases. > > From the information that you had shared, there are 2 more regression > use cases that I have in mind: > Use case#1: directory based regression Given that Autobuilder stored > result files inside /testresults directories, user shall be able to > perform the directory based regression using output from Autobuilder > directly, such as below Autobuilder directories. > https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/ > qemux86/testresults.json > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults > /qemux86/testresults.json > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults > /qemux86/testresults.json > > Assumed that there are 2 directories storing list of result files. > User shall provide these 2 directories for regression, regression > scripts will first parse through all the available files inside each > directories, then perform regression based on available configuration > data to determine the regression pair (eg. select result_set_1 from > directory#1 and result_set_x from directory#2 if they both have > matching configurations). Yes, this would be very useful. I suspect you don't need to have matching layouts, just import from all the json files in a given directory for the comparison. This way we can support arbitrary layouts. > Use case#2: git branch based regression Given that Autobuilder stored > result files inside /testresults directories, user shall first store > these directories and the result files in each git branch accordingly > using the existing store plugin. > After that, user can used the git branch based regression to analysis > the information. > Store in yocto-2.6.1.rc1, yocto-2.7_M1.rc1, yocto-2.7_M2.rc1 git > branch accordingly > https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/ > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults > / > https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults > / > > Assumed that result files are stored inside git repository with > specific git branch storing result files for single commit. User shall > provide the 2 specific git branches for regression, regression scripts > will first parse through all the available files inside each git > branch, then perform regression based on available configuration data > to determine the regression pair (eg. select result_set_1 from > git_branch_1 and result_set_x from git_branch_2 if they both have > matching configurations). > > The current codebase can be easily extended to enable both use cases > above. Please let me know if both use cases above are important and > please give us your inputs. Yes, I think these are two key use cases we need to support. I think there is a third thing we also need to look at: It would be great if there was some way of allowing some kind of templating when storing into the git repository. This way a general local log file from tmp/log/oeqa could be stored into the git repo, being split according to the layout of the repo if needed. Our default layout could match that from the autobuilder but the repository could define a layout? As mentioned, we also need to think about ptest. Currently the runtime ptest code has some parsing and log generation code. I think pieces like: http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/lib/oeqa/utils/logparser.py#n101 and the log_as_files() code should be moved to the reporting tool and that the test code should just generate the json results file which can later be parsed/processed as needed. I did post on the list earlier about some of the other challenges we have with the ptest data. buildperf is the other piece of this which we'll need to think about. Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
