Right, performance is the major reason chrome new headless will not be used for the majority of WPTs. One mitigation plan is to run all WPTs on CI with chrome new headless, where it is less sensitive to the total test time.
On Wed, Sep 4, 2024 at 4:47 AM PhistucK <phist...@gmail.com> wrote: > Sounds like headless shell will be used for the majority of the tests, but > that is not testing the full browser, but just a headless shell which goes > through different code paths... > I realise that previously it was probably run using content shell or > something like that which is also not as representative, are there any > plans to actually use the full browser for running tests? > If it is too slow, well... File bugs for making it faster ;) > > ☆*PhistucK* > > > On Tue, Sep 3, 2024 at 7:10 PM 'Weizhong Xia' via blink-dev < > blink-dev@chromium.org> wrote: > >> TLDR: chrome_wpt_tests and headless_shell_wpt_tests have been added to >> Chromium CI/CQ, and migration to Wptrunner has started. >> >> Hello there >> >> For quite some time web platform developers have wished to have a uniform >> way to run web platform tests, which is to converge to Wptrunner. Blink >> Engprod team has been working on this for quite some time, and now we are >> happy to announce that the infra is ready and we can now start to migrate >> the WPTs to run with Wptrunner, or run_wpt_tests.py, which is a wrapper >> built on top of Wptrunner and aims to make it easier to use Wptrunner in >> chromium. >> >> Migration Plan >> >> To migrate a directory to Wptrunner, you could simply add the directory >> to //third_party/blink/web_tests/TestLists/headless_shell.filter/ >> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/TestLists/headless_shell.filter>, >> and the tests in that directory will start to run with Wptrunner + Headless >> shell on Chromium CI/CQ. Our plan is to fully migrate to Wptrunner by the >> *end >> of the year*. By that time we will have three test suites for web tests, >> as listed in the table below: >> >> Suite Name >> >> Browser Under Test >> >> Harness >> >> Tests Run >> >> blink_web_tests >> >> Content Shell >> >> run_web_tests.py >> >> Legacy web tests and wpt_internal >> >> chrome_wpt_tests >> >> Chrome new headless >> >> run_wpt_tests.py >> >> WPTs under external/wpt that depends on //chrome layer >> >> headless_shell_wpt_tests >> >> Headless Shell >> >> run_wpt_tests.py >> >> All other WPTs under external/wpt >> >> How could you help? >> >> The migration will be done directory by directory, and there are 2000+ >> regressions on each platform. We will start to create crbugs with the label >> “wptrunner-migration” for those regressions. Please watch out for the >> crbugs and help fix them so that we can finish the migration on time. >> >> If you need any support during this process, the document is here >> <https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/run_web_platform_tests.md>. >> Also please feel free to reach out to weizhong@ or jonathanjlee@ if you >> want to have a chat. >> >> Thank you! >> >> Weizhong on behalf of Blink Engprod >> >> -- >> Empower web platform community with a single test runner that is fast, >> stable and easy to use. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "blink-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to blink-dev+unsubscr...@chromium.org. >> To view this discussion on the web visit >> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirCEhXLO%3Dz%3DnhwCpjkfcNZ0Q8Ox-gnOBytUURp8OGReNg%40mail.gmail.com >> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirCEhXLO%3Dz%3DnhwCpjkfcNZ0Q8Ox-gnOBytUURp8OGReNg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiqZLOYC3AUKUPRkdMp8jGERx2dvDW6CeAqxVzdfy5biCQ%40mail.gmail.com.
