On Thu, 2026-05-14 at 15:41 -0400, Trevor Gamblin wrote:
> Make changes to the patchtest script, helper modules, test suites, and
> selftest
> files to address various shortcomings with the current version, including:
>
> - Re-enable the previously included mergeability tests, which have been broken
> since prior to the 2023 Yocto RFQ
> - Improve series testing by fixing signal handling and return code logic for
> the
> '--directory' flag, and by making it possible for patchtest to report the
> shortlog for failing commits when testing series as single files
> - Make branch checking for merge tests more robust but retain flexibility for
> the selftest suite, and so that users can test series against branches other
> than the target one
> - Add new tests for malformed lines and whitespace in the TestPatch module
> - Refactors across the entire suite to remove dead code, simplify logic, and
> update selftest files
> - Expand the selftest file suite for all of the new cases added, bringing us
> from 38 to 51 selftests.
>
> Before:
>
> > ============================================================================
> > Testsuite summary for patchtest
> > ============================================================================
> > # TOTAL: 38
> > # XPASS: 18
> > # XFAIL: 18
> > # XSKIP: 2
> > # PASS: 0
> > # FAIL: 0
> > # SKIP: 0
> > # ERROR: 0
> > ============================================================================
>
> After:
>
> > ============================================================================
> > Testsuite summary for patchtest
> > ============================================================================
> > # TOTAL: 51
> > # XPASS: 25
> > # XFAIL: 26
> > # XSKIP: 0
> > # PASS: 0
> > # FAIL: 0
> > # SKIP: 0
> > # ERROR: 0
> > ============================================================================
>
> As always, there's lots of room for patchtest to grow further (I'd still very
> much like to convert it to a standalone Python tool one day). However, this is
> already a sizeable rework, and it should bring immediate benefit to anyone
> using
> it. I've made a point to label each individual patch appropriately, but for
> emphasis this work has been done in tandem with the Claude AI, specifically
> Sonnet 4.6 and 4.7.
>
> Given the skepticism that many of us have towards AI tooling, I made a point
> to
> do a review of the changes Claude had made myself before performing the final
> squash and rebase. This did reveal some bloat that I've since removed, but for
> the most part it made functional and useful changes. The selftest suite was
> routinely triggered locally to ensure it didn't cause any bizarre regressions,
> and new selftest files were added immediately for new test cases (or, in the
> case of the mergeability tests, refactored so that they were no longer labeled
> as skips). I've also made a point to pull patch series at random from the
> mailing list and test directly.
>
> Parts of this series (such as the '--directory' usage improvements) were
> written
> to address the needs of folks at Smile. Others address one or more outstanding
> bugs in Bugzilla, which I haven't consistently labeled as such (but I'll close
> them when this gets merged). The next priority will be to improve the actual
> deployment of the Patchtest service on the Yocto Project infrastructure.
>
> Trevor Gamblin (11):
> patchtest: check for meta-selftest, cleanup script
> patchtest: test series mergeability
> patchtest: tests: cleanup test suites
> patchtest: correctly abort --directory test
> patchtest: fix branch applicability reporting
> patchtest: test_patch: improve patch-specific tests
> patchtest: test_mbox: skip cover letters
> patchtest: mbox.py: improve 'From' line matching
> patchtest: selftest: stop if patchtest fails
> patchtest: selftest: rework test files
> patchtest: selftest: extend test file suite
>
> meta/lib/patchtest/mbox.py | 56 ++---
> meta/lib/patchtest/patchtest_parser.py | 2 +-
> meta/lib/patchtest/patchtest_patterns.py | 5 +
> meta/lib/patchtest/repo.py | 48 +++++
> ...x.pretest_series_applies_on_branch.1.pass} | 22 +-
> ...x.pretest_series_applies_on_branch.2.fail} | 0
> .../TestMbox.test_bugzilla_entry_format.fail | 5 +-
> .../TestMbox.test_bugzilla_entry_format.pass | 5 +-
> ...TestMbox.test_commit_message_presence.fail | 5 +-
> ...TestMbox.test_commit_message_presence.pass | 5 +-
> ...tMbox.test_commit_message_user_tags.1.pass | 5 +-
> ...tMbox.test_commit_message_user_tags.2.pass | 5 +-
> ...estMbox.test_commit_message_user_tags.fail | 5 +-
> .../files/TestMbox.test_mbox_format.pass | 5 +-
> .../files/TestMbox.test_non_auh_upgrade.fail | 25 +++
> .../files/TestMbox.test_non_auh_upgrade.pass | 25 +++
> .../files/TestMbox.test_shortlog_format.fail | 5 +-
> .../files/TestMbox.test_shortlog_format.pass | 5 +-
> .../files/TestMbox.test_shortlog_length.fail | 5 +-
> .../files/TestMbox.test_shortlog_length.pass | 5 +-
> ...estMbox.test_signed_off_by_presence.1.fail | 5 +-
> ...estMbox.test_signed_off_by_presence.2.fail | 5 +-
> .../TestMbox.test_signed_off_by_presence.pass | 5 +-
> .../TestMbox.test_target_mailing_list.1.fail | 26 +++
> .../TestMbox.test_target_mailing_list.2.fail | 24 +++
> .../TestMbox.test_target_mailing_list.pass | 25 +++
> .../TestMetadata.test_max_line_length.fail | 26 +++
> .../TestMetadata.test_max_line_length.pass | 25 +++
> .../files/TestPatch.test_cve_tag_format.fail | 5 +-
> .../files/TestPatch.test_cve_tag_format.pass | 5 +-
> ...estPatch.test_patch_malformed_context.fail | 38 ++++
> ...estPatch.test_patch_malformed_context.pass | 38 ++++
> ...tPatch.test_patch_trailing_whitespace.fail | 38 ++++
> ...tPatch.test_patch_trailing_whitespace.pass | 38 ++++
> ...TestPatch.test_signed_off_by_presence.fail | 5 +-
> ...TestPatch.test_signed_off_by_presence.pass | 5 +-
> ....test_upstream_status_presence_format.fail | 61 ++++++
> ....test_upstream_status_presence_format.pass | 62 ++++++
> meta/lib/patchtest/selftest/selftest | 25 ++-
> meta/lib/patchtest/tests/base.py | 117 ++++------
> meta/lib/patchtest/tests/test_mbox.py | 56 +++--
> meta/lib/patchtest/tests/test_metadata.py | 112 +++++-----
> meta/lib/patchtest/tests/test_patch.py | 113 ++++++++--
> .../lib/patchtest/tests/test_python_pylint.py | 8 +-
> scripts/patchtest | 200 +++++++++---------
> 45 files changed, 904 insertions(+), 406 deletions(-)
> rename
> meta/lib/patchtest/selftest/files/{TestMbox.test_series_merge_on_head.1.skip
> => TestMbox.pretest_series_applies_on_branch.1.pass} (66%)
> rename
> meta/lib/patchtest/selftest/files/{TestMbox.test_series_merge_on_head.2.skip
> => TestMbox.pretest_series_applies_on_branch.2.fail} (100%)
> create mode 100644
> meta/lib/patchtest/selftest/files/TestMbox.test_non_auh_upgrade.fail
> create mode 100644
> meta/lib/patchtest/selftest/files/TestMbox.test_non_auh_upgrade.pass
> create mode 100644
> meta/lib/patchtest/selftest/files/TestMbox.test_target_mailing_list.1.fail
> create mode 100644
> meta/lib/patchtest/selftest/files/TestMbox.test_target_mailing_list.2.fail
> create mode 100644
> meta/lib/patchtest/selftest/files/TestMbox.test_target_mailing_list.pass
> create mode 100644
> meta/lib/patchtest/selftest/files/TestMetadata.test_max_line_length.fail
> create mode 100644
> meta/lib/patchtest/selftest/files/TestMetadata.test_max_line_length.pass
> create mode 100644
> meta/lib/patchtest/selftest/files/TestPatch.test_patch_malformed_context.fail
> create mode 100644
> meta/lib/patchtest/selftest/files/TestPatch.test_patch_malformed_context.pass
> create mode 100644
> meta/lib/patchtest/selftest/files/TestPatch.test_patch_trailing_whitespace.fail
> create mode 100644
> meta/lib/patchtest/selftest/files/TestPatch.test_patch_trailing_whitespace.pass
> create mode 100644
> meta/lib/patchtest/selftest/files/TestPatch.test_upstream_status_presence_format.fail
> create mode 100644
> meta/lib/patchtest/selftest/files/TestPatch.test_upstream_status_presence_format.passHi Trevor, This is a difficult series to review, there are a lot of changes and several of the patches do more than one thing. Patches 1 & 2 are especially busy, and patch 6 both adds new tests and fixes existing tests - these should be separated into multiple commits that each do one thing. Is it possible to break this apart into a couple of separate patch series? That would make review significantly easier. Perhaps start with an initial series which just includes bugfixes and cleanup of existing test cases, then once that is merged send a second series which adds new test cases. Best regards, -- Paul Barker
signature.asc
Description: This is a digitally signed message part
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#237204): https://lists.openembedded.org/g/openembedded-core/message/237204 Mute This Topic: https://lists.openembedded.org/mt/119319575/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
