The patches from https://bugzilla.mozilla.org/show_bug.cgi?id=1975980 to reset modified preferences between browser-chrome mochitests have been merged to mozilla-central. This is a big step towards ensuring browser-chrome tests can run independently from each other, and making Test Verify jobs tier 1. This also makes the behavior of browser chrome mochitests closer to the behavior of plain mochitests that were already resetting preferences between tests.
To make this possible, a few preferences that were modified during many tests (but not by the tests themselves) were added to https://searchfox.org/mozilla-central/source/testing/mochitest/ignorePrefs.json and several tests that used to depend on preferences set by previous tests of the same manifest were fixed. Before these recent changes, preference changes caused unexpected failures in Test Verify jobs (since https://bugzilla.mozilla.org/show_bug.cgi?id=1816549), and were completely silent in other jobs (locally you could see them by adding the --compare-preferences command line parameter), which caused surprises when people were needinfo’ed on bugs opened for the TV failures. Now the preference changes are reported as expected failures on all jobs. On treeherder, these expected failure messages won’t turn the jobs orange, but if an unexpected failure occurs, both unexpected and expected failure messages will be visible. Test jobs upload a modifiedPrefs.json artifact containing the list of preferences that were reset for each test, making debugging easier. A script attached to the bug can be used to fetch the modifiedPrefs.json artifacts of an entire push and summarize them in an HTML report. For example https://bug1975980.bmoattachments.org/attachment.cgi?id=9500935 shows 316 prefs modified by 572 tests for a full mozilla-central push today. When running tests locally, you will see messages like: ERROR TEST-EXPECTED-FAIL | browser/base/content/test/webextensions/browser_permissions_dismiss.js | changed preference: extensions.webextensions.uuids There is no need to fix these errors urgently, but if you see tests in your component leaving prefs behind without a good reason, please fix them. Eventually (after a lot more cleanup happens) we would like to make preference changes unexpected failures so we stop introducing more. -- Florian Quèze -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAFu36Sq1jFeHS%2BpURQobWuNHnDReD5piDDshyd8yYKqsV66ecg%40mail.gmail.com.
