On 23/03/2022 23:47, Chris Johns wrote:> On 24/3/2022 3:38 am, Sebastian
Huber wrote:
I plan to commit the new validation tests tomorrow. Then I will make some test
runs. The test reports sent to bu...@rtems.org have only limited value to track
performance regressions for example.
The test reports sent to that list are archived and could be post processed as a
batch. A procmail filter could then be used to then maintain any batch generated
data. So there is some flexibility with its data.
A mailing list is nice because it is cheap and easy to create tools to send
reports.
You could also send the YAML report as a patch e-mail.
Would it make sense to store the full test
reports in YAML format in an RTEMS repository (for example
rtems-central/test-reports/arch/bsp/yyyy-mm-dd-commit.yml)?
It would be ideal having in a release the YAML and human digestible reports for
all tier 1 archs. I am not sure that can be achieved because the release's hash
and the tested hash may differ and we do not have a central hardware lab we can
access and use to create the reports when creating the release.
This is not only useful for releases. The time when you run tests is
somewhat arbitrary. Having the full raw test data available can be used
to figure out known good versions of a BSP in case something breaks.
This helps during a git bisect. It can be also used to gather
performance data which can be compared over time to notice regressions
introduced by new compiler versions and the RTEMS development. It is
important that only test reports are added which use commits of RTEMS
repositories without any local modifications so that everyone is able to
reproduce the data.
I am not a fan of adding dated files like this to repos. Every time I have done
this I have come to regret it. What about a path on ftp.rtems.org, for example
pub/rtems/reports/validation/6/arch/bsp/yyyy-mm-dd-commit.[yml,pdf] ?
What archs will be supported?
The YAML report is supported by rtems-test in general.
Is there a PDF type report?
For archiving a good human and machine readable format like YAML is
better suited. You can even diff two reports easily.
--
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.hu...@embedded-brains.de
phone: +49-89-18 94 741 - 16
fax: +49-89-18 94 741 - 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
