On Fri, 2023-01-13 at 16:06 +0100, Alexis Lothoré via lists.yoctoproject.org wrote: > This work has been initiated while discussing issue #14065 > (https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065) > The topic initially covers ptest reports between releases, but after > discussions > with R. Purdie > (https://lists.yoctoproject.org/g/automated-testing/message/1211), > a general regression report reliability topic emerged. Those first > patches aims to make regression reports more useful during releases by > computing > against which revision it would be relevant to check for regressions. > > A few notes about those patches : > - Since it affects release process, I struggled to properly test the new > behavior, > so testing has been made by stubbing send-qa-email side effects and picking > some > samples of build properties and layerinfo.json from autobuilder history to run > manually send-qa-email on my machine. But I will gladly take into account any > advice or procedure to test it more thoroughly > - I also have some uncommitted python unit tests on "previous version > computation", but current scripts directory does not allow proper unit tests > integration. If appropriate, I could submit some minor reworks to make it > viable > for testing and submit those tests
I'm glad you mentioned this! Yes, I think it is important we add some tests for this. We can rework these scripts as needed as the project is their main user at this point. > - This new regression checking introduced the need to clone a wider > testresults > history, which is (very) slow on my machine, but I do not now how slow it is > on autobuilder. Could it be an issue for the delivery process ? Does it need > to > be smarter on this point ? I think it will be slow on the autobuilder and we're going to have to do something different to make this work well. My clone still hasn't finished and is at 5.75GB of data now (15 mins to get the last 4.5GB) and still going. > - If appropriate, new patch could follow to allow regression reports > generation > against multiple revisions : for example we could check for a milestone build > against previous milestone AND previous major release I think that is the direction we need to go so yes please! Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#58983): https://lists.yoctoproject.org/g/yocto/message/58983 Mute This Topic: https://lists.yoctoproject.org/mt/96247368/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
