As a further example of this I wanted to experiment with a real world example. We have:
https://autobuilder.yocto.io/pub/releases/yocto-2.6.1.rc1/testresults/qemux86/testresults.json https://autobuilder.yocto.io/pub/releases/yocto-2.7_M1.rc1/testresults/qemux86/testresults.json https://autobuilder.yocto.io/pub/releases/yocto-2.7_M2.rc1/testresults/qemux86/testresults.json which are all results files for the same thing from 2.6.1, 2.7M1 and 2.7M2. I also have one for 2.6.0 locally but its the same as 2.6.1. Running this gives: resultstool regression qemux86-2.6.json qemux86-2.7-M2.json Successfully loaded base test results from: qemux86-2.6.json Successfully loaded target test results from: qemux86-2.7-M2.json Getting regression for base=runtime_core-image-minimal_qemux86_20181029181546 target=runtime_core-image-minimal_qemux86_20190124185230 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== ==============================End Regression============================== Getting regression for base=runtime_core-image-sato-sdk_qemux86_20181029181546 target=runtime_core-image-sato-sdk_qemux86_20190124185230 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== ==============================End Regression============================== Getting regression for base=runtime_core-image-sato_qemux86_20181029181546 target=runtime_core-image-sato_qemux86_20190124185230 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== ==============================End Regression============================== Getting regression for base=sdk_core-image-sato_i686_qemux86_20181029190813 target=sdk_core-image-sato_i686_qemux86_20190124185230 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== python.PythonTest.test_python3 : UNKNOWN -> None ==============================End Regression============================== Getting regression for base=sdk_core-image-sato_x86_64_qemux86_20181029192305 target=sdk_core-image-sato_x86_64_qemux86_20190124192316 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== python.PythonTest.test_python3 : UNKNOWN -> None ==============================End Regression============================== Getting regression for base=sdkext_core-image-minimal_x86_64_qemux86_20181029192305 target=sdkext_core-image-minimal_x86_64_qemux86_20190124192316 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== python.PythonTest.test_python3 : PASSED -> None ==============================End Regression============================== Getting regression for base=sdkext_core-image-sato_x86_64_qemux86_20181029192305 target=sdkext_core-image-sato_x86_64_qemux86_20190124192316 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== python.PythonTest.test_python3 : PASSED -> None ==============================End Regression============================== which shows that: a) In the case of no regressions it may be better simply print that rather than a header and empty body b) It does highlight that the "python3" test somehow disappeared c) Its hard to know that in fact another python3 test started passing. Maybe we also need a compare command to show "new" results to make it easier to investigate these things? For interest I ran the comparision backwards: resultstool regression qemux86-2.7-M2.json qemux86-2.6.json Successfully loaded base test results from: qemux86-2.7-M2.json Successfully loaded target test results from: qemux86-2.6.json Getting regression for base=runtime_core-image-minimal_qemux86_20190124185230 target=runtime_core-image-minimal_qemux86_20181029181546 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== apt.AptRepoTest.test_apt_install_from_repo : SKIPPED -> None buildcpio.BuildCpioTest.test_cpio : SKIPPED -> None buildgalculator.GalculatorTest.test_galculator : SKIPPED -> None buildlzip.BuildLzipTest.test_lzip : SKIPPED -> None connman.ConnmanTest.test_connmand_help : SKIPPED -> None connman.ConnmanTest.test_connmand_running : SKIPPED -> None date.DateTest.test_date : SKIPPED -> None df.DfTest.test_df : SKIPPED -> None dnf.DnfBasicTest.test_dnf_help : SKIPPED -> None dnf.DnfBasicTest.test_dnf_history : SKIPPED -> None dnf.DnfBasicTest.test_dnf_info : SKIPPED -> None dnf.DnfBasicTest.test_dnf_search : SKIPPED -> None dnf.DnfBasicTest.test_dnf_version : SKIPPED -> None dnf.DnfRepoTest.test_dnf_exclude : SKIPPED -> None dnf.DnfRepoTest.test_dnf_install : SKIPPED -> None dnf.DnfRepoTest.test_dnf_install_dependency : SKIPPED -> None dnf.DnfRepoTest.test_dnf_install_from_disk : SKIPPED -> None dnf.DnfRepoTest.test_dnf_install_from_http : SKIPPED -> None dnf.DnfRepoTest.test_dnf_installroot : SKIPPED -> None dnf.DnfRepoTest.test_dnf_makecache : SKIPPED -> None dnf.DnfRepoTest.test_dnf_reinstall : SKIPPED -> None dnf.DnfRepoTest.test_dnf_repoinfo : SKIPPED -> None gcc.GccCompileTest.test_gcc_compile : SKIPPED -> None gcc.GccCompileTest.test_gpp2_compile : SKIPPED -> None gcc.GccCompileTest.test_gpp_compile : SKIPPED -> None gcc.GccCompileTest.test_make : SKIPPED -> None gi.GObjectIntrospectionTest.test_python : SKIPPED -> None kernelmodule.KernelModuleTest.test_kernel_module : SKIPPED -> None ldd.LddTest.test_ldd : SKIPPED -> None logrotate.LogrotateTest.test_1_logrotate_setup : SKIPPED -> None logrotate.LogrotateTest.test_2_logrotate : SKIPPED -> None oe_syslog.SyslogTest.test_syslog_running : SKIPPED -> None oe_syslog.SyslogTestConfig.test_syslog_logger : SKIPPED -> None oe_syslog.SyslogTestConfig.test_syslog_restart : SKIPPED -> None oe_syslog.SyslogTestConfig.test_syslog_startup_config : SKIPPED -> None opkg.OpkgRepoTest.test_opkg_install_from_repo : SKIPPED -> None pam.PamBasicTest.test_pam : SKIPPED -> None parselogs.ParseLogsTest.test_parselogs : SKIPPED -> None perl.PerlTest.test_perl_works : SKIPPED -> None ptest.PtestRunnerTest.test_ptestrunner : SKIPPED -> None python.PythonTest.test_python3 : SKIPPED -> None rpm.RpmBasicTest.test_rpm_help : SKIPPED -> None rpm.RpmBasicTest.test_rpm_query : SKIPPED -> None rpm.RpmInstallRemoveTest.test_check_rpm_install_removal_log_file_size : SKIPPED -> None rpm.RpmInstallRemoveTest.test_rpm_install : SKIPPED -> None rpm.RpmInstallRemoveTest.test_rpm_query_nonroot : SKIPPED -> None rpm.RpmInstallRemoveTest.test_rpm_remove : SKIPPED -> None scp.ScpTest.test_scp_file : SKIPPED -> None ssh.SSHTest.test_ssh : SKIPPED -> None stap.StapTest.test_stap : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_basic : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_failed : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_list : SKIPPED -> None systemd.SystemdJournalTests.test_systemd_boot_time : SKIPPED -> None systemd.SystemdJournalTests.test_systemd_journal : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_disable_enable : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_status : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_stop_start : SKIPPED -> None xorg.XorgTest.test_xorg_running : SKIPPED -> None ==============================End Regression============================== Getting regression for base=runtime_core-image-sato-sdk_qemux86_20190124185230 target=runtime_core-image-sato-sdk_qemux86_20181029181546 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== apt.AptRepoTest.test_apt_install_from_repo : SKIPPED -> None opkg.OpkgRepoTest.test_opkg_install_from_repo : SKIPPED -> None pam.PamBasicTest.test_pam : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_basic : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_failed : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_list : SKIPPED -> None systemd.SystemdJournalTests.test_systemd_boot_time : SKIPPED -> None systemd.SystemdJournalTests.test_systemd_journal : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_disable_enable : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_status : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_stop_start : SKIPPED -> None ==============================End Regression============================== Getting regression for base=runtime_core-image-sato_qemux86_20190124185230 target=runtime_core-image-sato_qemux86_20181029181546 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== apt.AptRepoTest.test_apt_install_from_repo : SKIPPED -> None buildcpio.BuildCpioTest.test_cpio : SKIPPED -> None buildgalculator.GalculatorTest.test_galculator : SKIPPED -> None buildlzip.BuildLzipTest.test_lzip : SKIPPED -> None gcc.GccCompileTest.test_gcc_compile : SKIPPED -> None gcc.GccCompileTest.test_gpp2_compile : SKIPPED -> None gcc.GccCompileTest.test_gpp_compile : SKIPPED -> None gcc.GccCompileTest.test_make : SKIPPED -> None kernelmodule.KernelModuleTest.test_kernel_module : SKIPPED -> None ldd.LddTest.test_ldd : SKIPPED -> None logrotate.LogrotateTest.test_1_logrotate_setup : SKIPPED -> None logrotate.LogrotateTest.test_2_logrotate : SKIPPED -> None opkg.OpkgRepoTest.test_opkg_install_from_repo : SKIPPED -> None pam.PamBasicTest.test_pam : SKIPPED -> None perl.PerlTest.test_perl_works : PASSED -> None stap.StapTest.test_stap : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_basic : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_failed : SKIPPED -> None systemd.SystemdBasicTests.test_systemd_list : SKIPPED -> None systemd.SystemdJournalTests.test_systemd_boot_time : SKIPPED -> None systemd.SystemdJournalTests.test_systemd_journal : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_disable_enable : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_status : SKIPPED -> None systemd.SystemdServiceTests.test_systemd_stop_start : SKIPPED -> None ==============================End Regression============================== Getting regression for base=sdk_core-image-sato_i686_qemux86_20190124185230 target=sdk_core-image-sato_i686_qemux86_20181029190813 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== buildepoxy.EpoxyTest.test_epoxy : PASSED -> None python.Python2Test.test_python2 : SKIPPED -> None python.Python3Test.test_python3 : PASSED -> None ==============================End Regression============================== Getting regression for base=sdk_core-image-sato_x86_64_qemux86_20190124192316 target=sdk_core-image-sato_x86_64_qemux86_20181029192305 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== buildepoxy.EpoxyTest.test_epoxy : PASSED -> None python.Python2Test.test_python2 : SKIPPED -> None python.Python3Test.test_python3 : PASSED -> None ==============================End Regression============================== Getting regression for base=sdkext_core-image-minimal_x86_64_qemux86_20190124192316 target=sdkext_core-image-minimal_x86_64_qemux86_20181029192305 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== buildepoxy.EpoxyTest.test_epoxy : SKIPPED -> None python.Python2Test.test_python2 : SKIPPED -> None python.Python3Test.test_python3 : SKIPPED -> None ==============================End Regression============================== Getting regression for base=sdkext_core-image-sato_x86_64_qemux86_20190124192316 target=sdkext_core-image-sato_x86_64_qemux86_20181029192305 ============================Start Regression============================ Only print regression if base status not equal target <test case> : <base status> -> <target status> ======================================================================== buildepoxy.EpoxyTest.test_epoxy : SKIPPED -> None python.Python2Test.test_python2 : SKIPPED -> None python.Python3Test.test_python3 : SKIPPED -> None ==============================End Regression============================== and this shows there were a number of things which started showing up in the test results files, mostly more skips being logged which is understandable given the chanegs we made in recent versions of the QA code. This does have a real world use in that we have been trying to figure out if the ADT test result changes in 2.6.1 are a regression. Given the data I've seen, I suspect the answer is now "no, its not a regression, we just have some test improvements in 2.7 which we may want to backport". Cheers, Richard -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
