It has been observed that useful information in regression report can be
drowned in huge regression lists which are often false-positives (for
example, a whole set of tests has been temporarily disabled). Additionally,
the regression report usually starts with not-so-useful information when
searching for regressions (basically, big lists of tests which have not
regressed).

This series aims to improve regression display by rearranging info order as
well as bringing a default limit to how many changes are displayed per
base/target comparison. This new version follows up suggestions discussed
in [1].

One consequence of this series is the new available option for
yocto_testresults_query:
- yocto_testresults_query 4.3_M1 4.3_M2
  -> will display at most 50 regressions per test type
- yocto_testresults_query -l 10 4.3_M1 4.3_M2
  -> override the display limit and reduce it to 10 regressions per test
     type
- yocto_testresults_query -l 0 4.3_M1 4.3_M2
  -> disable the display limit, print all regressions

An example of regression report with those modifications can be found at
[2]

Note: this series does not take care yet of improving base/target pairs
selection based on machine/distro (issue raised by Ross  in [1]),
because I still need to have a better look at this before tweaking anything
there.

Changes in v2:
- use the display limit for each test type instead of using it as a global
  limit
- display regressions summaries before detailed list
- display regressions first, display matched/improved tests after, as
  "additional data"
- Reduce less useful info size
- fix testresult selftests

[1] https://lore.kernel.org/yocto/90d7fe40-32eb-4d67-b099-9e0c764fc...@arm.com/
[2] https://pastebin.com/JHFLF0sK

Alexis Lothoré (4):
  scripts/resulttool: limit the number of changes displayed per test
  scripts/resulttool: rearrange regressions report order
  scripts/resulttool: make additional info more compact
  scripts/yocto_testresults_query: add option to change display limit

 scripts/lib/resulttool/regression.py | 60 +++++++++++++++++++++++-----
 scripts/yocto_testresults_query.py   | 13 ++++--
 2 files changed, 60 insertions(+), 13 deletions(-)

-- 
2.42.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189606): 
https://lists.openembedded.org/g/openembedded-core/message/189606
Mute This Topic: https://lists.openembedded.org/mt/102120368/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Reply via email to