[blink-dev] introduce blink_wpt_tests

['Weizhong Xia' via blink-dev]

Dear blink devs

As some of you may already know, I am working to split blink_web_tests into
two test suites: *blink_web_tests* for legacy tests and *blink_wpt_tests*
for wpt tests(crbug/1299834). We now have reached the first point to move
all baselines to a central place (CL
). Some
of the changes you may need to be aware of:
1. We renamed web_tests/platform to web_tests/baselines.
2. All the generic baselines are now put under web_tests/baselines/generic.
(reference files are allowed at both baselines/generic or in the same
folder of the test itself.)

I planned to do another round of rebaseline then land this CL the
coming Saturday, to reduce the chance of merge conflict.

After that, we will have another CL to actually break blink_web_tests into
two. This should have no impact on your local experiences. The only
difference is that we are now running legacy tests and wpt tests at
different steps, so you need to get log/results at different places.

At this point blink_wpt_tests will continue to run with run_web_tests.py.
We have a plan to switch it to use wptrunner, targeting the middle of next
year. After that we will have tools, web_tests, wpt_tests under blink, and
we will use metadata as test expectation for wpt_tests. More details to
come.

Thanks for your attention. If you have any suggestion/comments, feel free
to reply here, or put that in the crbug or CL. I definitely want to get
more input from you folks.

Thanks, Weizhong

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSioUQu-0wzBbQm1CG2ANTH_aPgdY1NuWS309zASAhcGjtg%40mail.gmail.com.

[blink-dev] Re: introduce blink_wpt_tests

['Weizhong Xia' via blink-dev]

We will still have two test suites for fuchsia, and filter the tests
properly from SmokeTests.

For the long term plan, I need to discuss with you folks (separately) on
how to run wptrunner for fuchsia.

On Mon, Apr 25, 2022 at 5:10 PM Zijie He  wrote:

> Weizhong, only SmokeTests
> 
> are running on fuchsia
> 
>  now
> by default. How would this split impact the smoke tests?
>
> On Monday, April 25, 2022 at 4:06:24 PM UTC-7 Weizhong Xia wrote:
>
>> Dear blink devs
>>
>> As some of you may already know, I am working to split blink_web_tests
>> into two test suites: *blink_web_tests* for legacy tests and
>> *blink_wpt_tests* for wpt tests(crbug/1299834 ).
>> We now have reached the first point to move all baselines to a central
>> place (CL
>> ).
>> Some of the changes you may need to be aware of:
>> 1. We renamed web_tests/platform to web_tests/baselines.
>> 2. All the generic baselines are now put under
>> web_tests/baselines/generic. (reference files are allowed at both
>> baselines/generic or in the same folder of the test itself.)
>>
>> I planned to do another round of rebaseline then land this CL the
>> coming Saturday, to reduce the chance of merge conflict.
>>
>> After that, we will have another CL to actually break blink_web_tests
>> into two. This should have no impact on your local experiences. The only
>> difference is that we are now running legacy tests and wpt tests at
>> different steps, so you need to get log/results at different places.
>>
>> At this point blink_wpt_tests will continue to run with run_web_tests.py.
>> We have a plan to switch it to use wptrunner, targeting the middle of next
>> year. After that we will have tools, web_tests, wpt_tests under blink, and
>> we will use metadata as test expectation for wpt_tests. More details to
>> come.
>>
>> Thanks for your attention. If you have any suggestion/comments, feel free
>> to reply here, or put that in the crbug or CL. I definitely want to get
>> more input from you folks.
>>
>> Thanks, Weizhong
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSioaNR4Udj3G-OPWrq5gSFF6u0Z4i0J6JXkeC%2BkFmF5%3DLw%40mail.gmail.com.

Re: [blink-dev] introduce blink_wpt_tests

['Weizhong Xia' via blink-dev]

I think it would be third_party/blink/wpt_tests/, and we will have
third_party/blink/wpt_tests/wpt_internal for internal tests. I would still
need to check if such structure could cause problems to the
importer/exporter.

On Tue, Apr 26, 2022 at 6:21 AM Harald Alvestrand  wrote:

> at the moment, wpt tests live under
> third_party/blink/web_tests/external/wpt (4 directories down).
>
> What will the new path for wpt tests be, once all is moved and done?
>
>
>
> On Tue, Apr 26, 2022 at 12:06 AM 'Weizhong Xia' via blink-dev <
> blink-dev@chromium.org> wrote:
>
>> Dear blink devs
>>
>> As some of you may already know, I am working to split blink_web_tests
>> into two test suites: *blink_web_tests* for legacy tests and
>> *blink_wpt_tests* for wpt tests(crbug/1299834 <https://crbug.com/1299834>).
>> We now have reached the first point to move all baselines to a central
>> place (CL
>> <https://chromium-review.googlesource.com/c/chromium/src/+/3603221>).
>> Some of the changes you may need to be aware of:
>> 1. We renamed web_tests/platform to web_tests/baselines.
>> 2. All the generic baselines are now put under
>> web_tests/baselines/generic. (reference files are allowed at both
>> baselines/generic or in the same folder of the test itself.)
>>
>> I planned to do another round of rebaseline then land this CL the
>> coming Saturday, to reduce the chance of merge conflict.
>>
>> After that, we will have another CL to actually break blink_web_tests
>> into two. This should have no impact on your local experiences. The only
>> difference is that we are now running legacy tests and wpt tests at
>> different steps, so you need to get log/results at different places.
>>
>> At this point blink_wpt_tests will continue to run with run_web_tests.py.
>> We have a plan to switch it to use wptrunner, targeting the middle of next
>> year. After that we will have tools, web_tests, wpt_tests under blink, and
>> we will use metadata as test expectation for wpt_tests. More details to
>> come.
>>
>> Thanks for your attention. If you have any suggestion/comments, feel free
>> to reply here, or put that in the crbug or CL. I definitely want to get
>> more input from you folks.
>>
>> Thanks, Weizhong
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "blink-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to blink-dev+unsubscr...@chromium.org.
>> To view this discussion on the web visit
>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSioUQu-0wzBbQm1CG2ANTH_aPgdY1NuWS309zASAhcGjtg%40mail.gmail.com
>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSioUQu-0wzBbQm1CG2ANTH_aPgdY1NuWS309zASAhcGjtg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirbrMKgfn3MsT_F%3DwUjNHumHupve5rLrPAkqT_3k9JY8w%40mail.gmail.com.

Re: [blink-dev] introduce blink_wpt_tests

['Weizhong Xia' via blink-dev]

The goal is to use wptrunner to run wpt tests. rwt will still be used to
run legacy tests. Do you imply you prefer to use rwt for both wpt tests and
legacy tests? I would love to hear your reasoning about that.

There are several business motivations behind that. To be able to run tests
on Android/iOS and reduce costs to developers are two of those.

On Tue, Apr 26, 2022 at 9:50 AM Christian Biesinger 
wrote:

> Will we now/later need to start using wptrunner instead of
> run_web_tests.py to run the tests locally?
>
> (BTW, what's the motivation for this change?)
>
> Christian
>
> On Mon, Apr 25, 2022 at 7:06 PM 'Weizhong Xia' via blink-dev
>  wrote:
> >
> > Dear blink devs
> >
> > As some of you may already know, I am working to split blink_web_tests
> into two test suites: blink_web_tests for legacy tests and blink_wpt_tests
> for wpt tests(crbug/1299834 <https://crbug.com/1299834>). We now have
> reached the first point to move all baselines to a central place (CL). Some
> of the changes you may need to be aware of:
> > 1. We renamed web_tests/platform to web_tests/baselines.
> > 2. All the generic baselines are now put under
> web_tests/baselines/generic. (reference files are allowed at both
> baselines/generic or in the same folder of the test itself.)
> >
> > I planned to do another round of rebaseline then land this CL the coming
> Saturday, to reduce the chance of merge conflict.
> >
> > After that, we will have another CL to actually break blink_web_tests
> into two. This should have no impact on your local experiences. The only
> difference is that we are now running legacy tests and wpt tests at
> different steps, so you need to get log/results at different places.
> >
> > At this point blink_wpt_tests will continue to run with
> run_web_tests.py. We have a plan to switch it to use wptrunner, targeting
> the middle of next year. After that we will have tools, web_tests,
> wpt_tests under blink, and we will use metadata as test expectation for
> wpt_tests. More details to come.
> >
> > Thanks for your attention. If you have any suggestion/comments, feel
> free to reply here, or put that in the crbug or CL. I definitely want to
> get more input from you folks.
> >
> > Thanks, Weizhong
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "blink-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to blink-dev+unsubscr...@chromium.org.
> > To view this discussion on the web visit
> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSioUQu-0wzBbQm1CG2ANTH_aPgdY1NuWS309zASAhcGjtg%40mail.gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipw9Cpfouy5X81anA8vVKAnGq8Rm64RPiZ1OSJw56r5cQ%40mail.gmail.com.

[blink-dev] Re: introduce blink_wpt_tests

['Weizhong Xia' via blink-dev]

Hi Folks

A follow up of this announcement. I was finally able to land this change
yesterday, with a small change. We later decided to keep the platform, to
reduce the size of the CL. So now all the baselines are under
*web_tests/platform*, and generic baselines are under
*web_tests/platform/generic*.

If you have any ongoing CL that changes baselines, pls make sure the
baselines are at the correct place. Feel free to ping me if you have any
questions.

thanks, Weizhong

On Mon, Apr 25, 2022 at 4:06 PM Weizhong Xia  wrote:

> Dear blink devs
>
> As some of you may already know, I am working to split blink_web_tests
> into two test suites: *blink_web_tests* for legacy tests and
> *blink_wpt_tests* for wpt tests(crbug/1299834 ).
> We now have reached the first point to move all baselines to a central
> place (CL
> ).
> Some of the changes you may need to be aware of:
> 1. We renamed web_tests/platform to web_tests/baselines.
> 2. All the generic baselines are now put under
> web_tests/baselines/generic. (reference files are allowed at both
> baselines/generic or in the same folder of the test itself.)
>
> I planned to do another round of rebaseline then land this CL the
> coming Saturday, to reduce the chance of merge conflict.
>
> After that, we will have another CL to actually break blink_web_tests into
> two. This should have no impact on your local experiences. The only
> difference is that we are now running legacy tests and wpt tests at
> different steps, so you need to get log/results at different places.
>
> At this point blink_wpt_tests will continue to run with run_web_tests.py.
> We have a plan to switch it to use wptrunner, targeting the middle of next
> year. After that we will have tools, web_tests, wpt_tests under blink, and
> we will use metadata as test expectation for wpt_tests. More details to
> come.
>
> Thanks for your attention. If you have any suggestion/comments, feel free
> to reply here, or put that in the crbug or CL. I definitely want to get
> more input from you folks.
>
> Thanks, Weizhong
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSir99Ng_Kb7XGAx-sC_EgyfCtLCgSHQ2E0XBV97rfTgzEw%40mail.gmail.com.

Re: [blink-dev] Developer pain as a result of "introducing blink_wpt_tests"

['Weizhong Xia' via blink-dev]

Folks

I'm sorry to see this has caused inconvenience, and sorry for being late in 
response to this, due to the same reason Dom had.

The reason to move baselines to one central place at this point is to make 
it possible to specify dependency in BUILD.gn. This is part of the work to 
use upstream wptrunner to run wpt tests. In crbug/1299834 we have tried to 
discuss some different approaches. I would say this is the least disruptive 
way. For blink engprod, I think to make blink devs happy is our top 
priority.

For the long term, we can definitely move this back once we have separate 
legacy tests and wpt into different folders. I can also reach out to you 
guys to better understand your needs.

For the folders Dom mentioned, I can check to see if any of that can be 
removed. I understand the README is needed for virtual test suites.

Cheers, Weizhong 

On Wednesday, June 1, 2022 at 6:06:56 AM UTC-7 yoav...@chromium.org wrote:

> +1 from me as well. I was similarly caught by surprise by this change 
> (during reviews for webexposed changes), and am similarly not seeing the 
> upside for this.
>
> While I'm sure this is a change that was meant to be a positive one, I'd 
> love to better understand the reasoning, and whether the current situation 
> is a temporary one, or one that is planned to be permanent even after the 
> move to blink_wpt_tests is done.
>
> On Fri, May 27, 2022 at 6:10 PM Domenic Denicola  
> wrote:
>
>> +1. This was really unpleasantly surprising. When I first saw the 
>> original blink-dev email, I thought "generic baselines" meant something 
>> like "non-web platform test baselines", not "WPT expectation files that are 
>> platform-agnostic".
>>
>> In addition to the context-switching cost, it's just much harder to 
>> navigate between tests and their expectations, which is something I do 
>> quite often. E.g., they are no longer grouped together in code reviews, 
>> since their file paths are lexicographically far away from each other. And, 
>> as someone maintaining and reviewing several WPT directories, moving these 
>> crucial files out of the directories I commonly work in (and have metadata 
>> marking me as the point-of-contact for) into separate directories dilutes 
>> the cohesiveness of my projects.
>>
>>
>> On Fri, May 27, 2022 at 12:03 PM Dominic Farolino  
>> wrote:
>>
>>> I write a lot of web platform tests as a Web Platform engineer; recently 
>>> I wrote one in external/wpt/ 
>>> 
>>>  
>>> (the external web platform tests directory), and was shocked to find the 
>>> eradication of `-expected.txt` files. I placed my expectations file next to 
>>> the source file as we've done for many years, and found that my test was 
>>> "failing" because the test runner couldn't find my test expectations file.
>>>
>>> I dug deeper and found https://crrev.com/c/3603221 which was 
>>> responsible for moving more than 21,000 *platform-agnostic* test 
>>> expectations files away from their source files and into 
>>> web_tests/platform/generic 
>>> 
>>>  
>>> directory. I found more discussion in this email thread 
>>>  which 
>>> I missed because blink-dev emails do not go directly in my inbox.
>>>
>>> I must say I find this change extraordinarily inconvenient as a Web 
>>> Platform engineer, and I want to push back against this. A minority of web 
>>> platform tests have platform-specific failures, which justifies the need 
>>> for *some* platform-specific test expectations directories, but I 
>>> believe a huge majority have generic baselines that are wildly convenient 
>>> to have right next to the actual tests themselves. Putting them in a 
>>> separate directory means I and others have to open a separate browser tab 
>>> to view how many expectations there are for a given directory, and requires 
>>> a lot of unnecessary context switching. It is particularly confusing for 
>>> clusters of tests whose names are all *very* similar and vary by only a 
>>> few numbers or suffixes—this increases the cost of the context switching.
>>>
>>> Furthermore, it renders tons of directories absolutely useless! All ~150 
>>> directories in web_tests/virtual 
>>> 
>>>  (for 
>>> VirtualTestSuites) are just empty directories with README files—these used 
>>> to house virtualtest-specific expectations. So now for fenced frames (the 
>>> project I'm working on right now), we have the following test directories:
>>>
>>>- web_tests/wpt_internal/fenced_frame/ 
>>>
>>> 
>>>- web_tests/virtual/fenced-frame-mparch/ 
>>>  

Re: [blink-dev] Developer pain as a result of "introducing blink_wpt_tests"

['Weizhong Xia' via blink-dev]

Previously you don't need to specify anything in BUILD.gn is because we are
downloading the whole "web_tests" folder. Now we want to run legacy tests
and wpt in different steps. Under "web_tests/virtual" we have baselines for
wpt and pure virtual legacy tests, we can list the folders under
"virtual/prefix" for each virtual test suite, which will make the
dependency very large(yes), and we need make such change each time when we
add new virtual suites. I don't think this is something blink devs want to
do.

The discussion is at the beginning of the crbug. So pls scroll back to #c1,
and read from there.

thanks, Weizhong

On Thu, Jun 2, 2022 at 12:29 PM Domenic Denicola 
wrote:

>
>
> On Thu, Jun 2, 2022 at 3:21 PM Weizhong Xia  wrote:
>
>> Folks
>>
>> I'm sorry to see this has caused inconvenience, and sorry for being late
>> in response to this, due to the same reason Dom had.
>>
>
> Thanks for responding and listening to our concerns!
>
>
>>
>> The reason to move baselines to one central place at this point is to
>> make it possible to specify dependency in BUILD.gn.
>>
>
> I don't quite understand this. I've had to work with WPTs and WPT
> expectations my entire time working on Chromium. I've never had to "specify
> dependency in BUILD.gn"; I don't really know what that means. I'd love to
> hear more (or be referred to a doc explaining the issue), so that I
> understand why we're making the sacrifice we're making. (The linked bug
> isn't very understandable for me, unfortunately. I can't even understand
> enough to find the part you mentioned where you discussed different
> approaches.)
>
> I'm also unsure how the decision was weighed. Is the population of people
> specifying dependency in BUILD.gn very large, so that their needs are
> outweighing those of the Chromium developers working with web platform
> tests?
>
>
>> This is part of the work to use upstream wptrunner to run wpt tests. In
>> crbug/1299834  we have tried to discuss some
>> different approaches. I would say this is the least disruptive way. For
>> blink engprod, I think to make blink devs happy is our top priority.
>>
>> For the long term, we can definitely move this back once we have separate
>> legacy tests and wpt into different folders. I can also reach out to you
>> guys to better understand your needs.
>>
>> For the folders Dom mentioned, I can check to see if any of that can be
>> removed. I understand the README is needed for virtual test suites.
>>
>> Cheers, Weizhong
>>
>> On Wednesday, June 1, 2022 at 6:06:56 AM UTC-7 yoav...@chromium.org
>> wrote:
>>
>>> +1 from me as well. I was similarly caught by surprise by this change
>>> (during reviews for webexposed changes), and am similarly not seeing the
>>> upside for this.
>>>
>>> While I'm sure this is a change that was meant to be a positive one, I'd
>>> love to better understand the reasoning, and whether the current situation
>>> is a temporary one, or one that is planned to be permanent even after the
>>> move to blink_wpt_tests is done.
>>>
>>> On Fri, May 27, 2022 at 6:10 PM Domenic Denicola 
>>> wrote:
>>>
 +1. This was really unpleasantly surprising. When I first saw the
 original blink-dev email, I thought "generic baselines" meant something
 like "non-web platform test baselines", not "WPT expectation files that are
 platform-agnostic".

 In addition to the context-switching cost, it's just much harder to
 navigate between tests and their expectations, which is something I do
 quite often. E.g., they are no longer grouped together in code reviews,
 since their file paths are lexicographically far away from each other. And,
 as someone maintaining and reviewing several WPT directories, moving these
 crucial files out of the directories I commonly work in (and have metadata
 marking me as the point-of-contact for) into separate directories dilutes
 the cohesiveness of my projects.


 On Fri, May 27, 2022 at 12:03 PM Dominic Farolino 
 wrote:

> I write a lot of web platform tests as a Web Platform engineer;
> recently I wrote one in external/wpt/
> 
> (the external web platform tests directory), and was shocked to find the
> eradication of `-expected.txt` files. I placed my expectations file next 
> to
> the source file as we've done for many years, and found that my test was
> "failing" because the test runner couldn't find my test expectations file.
>
> I dug deeper and found https://crrev.com/c/3603221 which was
> responsible for moving more than 21,000 *platform-agnostic* test
> expectations files away from their source files and into
> web_tests/platform/generic
> 
> directory. I found more discussion in this

Re: [blink-dev] Developer pain as a result of "introducing blink_wpt_tests"

['Weizhong Xia' via blink-dev]

Xianzhu, yes 'baselines' is the name we agreed on previously. The reason I
later changed back to use 'platform' is because that will make the CL
smaller, and make it easier for gerrit to handle it. We can make one round
rename when everything is stabilized. (I left you a message when you are
OOO. I am not sure if that message lived long enough for you to catch it.)

thanks, Weizhong

On Mon, Jun 6, 2022 at 3:30 PM Xianzhu Wang 
wrote:

> I think we first need to answer a question: Why do we need *-expected.txt
> for WPT tests?
>
> Upstream WPT doesn't have *-expected.txt. *-expected.txt is a
> blink-specific thing to allow some failing WPT tests to pass on blink with
> temporarily allowed failures. If a test has -expected.txt, it means Chrome
> behaves differently than the standard web platform behavior (or the test
> itself is wrong). The files are sometimes harmful because they hide
> failures and web platform incompatibilities (e.g.
> ). So I think -expected.txt files for WPT tests
> should be rare and eventually be removed. An -expected.txt should not be
> treated as a part of the test itself because a) it doesn't exist in
> upstream WPT and 2) it doesn't describe the standard expected behavior,
> thus isn't necessarily placed besides the test.
>
> The directory name 'platform' may be misleading. 'baselines' may be a
> better name (but we should not rename until we decide what to do for
> generic baselines). For WPT tests, 'failures' is perhaps an even better
> name.
>
> On Friday, June 3, 2022 at 6:55:49 AM UTC-7 Dominic Farolino wrote:
>
>> For the long term, we can definitely move this back once we have separate
>>> legacy tests and wpt into different folders. I can also reach out to you
>>> guys to better understand your needs.
>>>
>>> For the folders Dom mentioned, I can check to see if any of that can be
>>> removed. I understand the README is needed for virtual test suites.
>>>
>>
>> Can you define "long term" here? Is there a timeline? I was really hoping
>> that we'd go back to how things were almost immediately. Still the need for
>> the delay does not quite make sense to me, and I'm really hoping that for
>> Blink developer experience we can revert this back to the previous setup
>> ASAP.
>>
>> Under "web_tests/virtual" we have baselines for wpt and pure virtual
>>> legacy tests, we can list the folders under "virtual/prefix" for each
>>> virtual test suite, which will make the dependency very large(yes), and we
>>> need make such change each time when we add new virtual suites. I don't
>>> think this is something blink devs want to do.
>>>
>>
>> I'm sorry, but like Domenic I am not sure what to make of much of this.
>> What I do know is that Blink developers also don't want the current WPT
>> writing experience where expectation files are positioned far away from the
>> source test. I think this experience matters a lot.
>>
>> On Thu, Jun 2, 2022 at 4:25 PM Weizhong Xia  wrote:
>>
>>> Previously you don't need to specify anything in BUILD.gn is because we
>>> are downloading the whole "web_tests" folder. Now we want to run legacy
>>> tests and wpt in different steps. Under "web_tests/virtual" we have
>>> baselines for wpt and pure virtual legacy tests, we can list the folders
>>> under "virtual/prefix" for each virtual test suite, which will make the
>>> dependency very large(yes), and we need make such change each time when we
>>> add new virtual suites. I don't think this is something blink devs want to
>>> do.
>>>
>>> The discussion is at the beginning of the crbug. So pls scroll back to
>>> #c1, and read from there.
>>>
>>> thanks, Weizhong
>>>
>>> On Thu, Jun 2, 2022 at 12:29 PM Domenic Denicola 
>>> wrote:
>>>


 On Thu, Jun 2, 2022 at 3:21 PM Weizhong Xia 
 wrote:

> Folks
>
> I'm sorry to see this has caused inconvenience, and sorry for being
> late in response to this, due to the same reason Dom had.
>

 Thanks for responding and listening to our concerns!


>
> The reason to move baselines to one central place at this point is to
> make it possible to specify dependency in BUILD.gn.
>

 I don't quite understand this. I've had to work with WPTs and WPT
 expectations my entire time working on Chromium. I've never had to "specify
 dependency in BUILD.gn"; I don't really know what that means. I'd love to
 hear more (or be referred to a doc explaining the issue), so that I
 understand why we're making the sacrifice we're making. (The linked bug
 isn't very understandable for me, unfortunately. I can't even understand
 enough to find the part you mentioned where you discussed different
 approaches.)

 I'm also unsure how the decision was weighed. Is the population of
 people specifying dependency in BUILD.gn very large, so that their needs
 are outweighing those of the Chromium developers working with web platform
 tests?

Re: [blink-dev] Developer pain as a result of "introducing blink_wpt_tests"

['Weizhong Xia' via blink-dev]

Hi blink devs

Thanks to those who joined the survey at https://forms.gle/ju45qciS5VTR4ywN7.
Most of you expressed the desire to put baselines at the same place of the
tests. Your voice is heard, and here is the plan for the next step.

In Q3 we will work on to completely separate legacy layout tests and wpt
tests, to put them under `third_party/blink/web_tests` and
`third_party/blink/wpt_tests` respectively. Generic baselines (including
generic virtual baselines) will be moved back to their previous place.
Rebaseline tool will be updated to work with this structure, and update
baselines for legacy layout tests and wpt tests in a single run. We will
have different copies of *TestExpectations*, *FlagSpecificConfig*,
*VirtualTestSuites* etc for legacy tests and wpt tests. When working on
those files, we will need to make sure we are updating the correct copy of
the file. (We will investigate if we need some presubmit check for such a
scenario).

The reason for this is two fold: as requested we want to put baselines side
to side to the tests, and we want to make the directory structure right to
speed up the switch to wptrunner.

Thoughts? Feel free to leave a comment in crbug/1299834.

Thanks, Weizhong



On Mon, Jun 6, 2022 at 5:00 PM Xianzhu Wang 
wrote:

> On Mon, Jun 6, 2022 at 3:52 PM Weizhong Xia  wrote:
>
>> Xianzhu, yes 'baselines' is the name we agreed on previously. The reason
>> I later changed back to use 'platform' is because that will make the CL
>> smaller, and make it easier for gerrit to handle it. We can make one round
>> rename when everything is stabilized. (I left you a message when you are
>> OOO. I am not sure if that message lived long enough for you to catch it.)
>>
>
> I caught the message. I guessed that the name 'platform' might be one of
> the reasons for the surprise to blink developers after the change, and the
> name 'baselines' might make the change easier to explain :)
>
> On Mon, Jun 6, 2022 at 3:52 PM Domenic Denicola 
> wrote:
>
>> I'm not sure I understand this logic. -expected.txt files are just
>> more-convenient, more in-your-face versions of TestExpectations files.
>> Surely you're not suggesting we get rid of TestExpectations files?
>>
>
> Sorry, "-expected.txt ... should be ... eventually be removed" in my
> previous email was not clear. I meant for each individual -expected.txt we
> should eventually remove it because we should fix the failure. The same
> logic applies to TestExpectations. At any time we may allow a certain
> number of failures but we should keep the number as small as possible.
>
> I think we should prefer TestExpectations to -expected.txt for WPT tests
> because the entries in TestExpectations have associated bugs which track
> the fixing process, unless we find a better way to track the fixing of the
> failures in -expected.txt. -expected.txt files do have their values, e.g.
> for partially-passing tests we can discover regressions and progressions of
> individual sub tests, but they should be rare.
>
> I think separating -expected.txt from the tests has the following benefits:
> - It makes it clear to blink developers that the files are not a part of
> WPT.
> - It simplifies the WPT export/import process and others by reducing
> blink-specific files under external/wpt.
>
> It does make it more difficult to find -expected.txt, but we already have
> the similar well-known logic for platform-specific baselines. Though
> platform-specific baselines are rare, ignoring a platform baseline can
> still cause surprises.
>
> I think we can improve the test result viewer
> 
> - to better show -expected.txt for passing tests
> - to show information about tests without actually running the tests
> WDYT?
>
>
>>
>> On Mon, Jun 6, 2022 at 6:30 PM Xianzhu Wang 
>> wrote:
>>
>>> I think we first need to answer a question: Why do we need
>>> *-expected.txt for WPT tests?
>>>
>>> Upstream WPT doesn't have *-expected.txt. *-expected.txt is a
>>> blink-specific thing to allow some failing WPT tests to pass on blink with
>>> temporarily allowed failures. If a test has -expected.txt, it means Chrome
>>> behaves differently than the standard web platform behavior (or the test
>>> itself is wrong). The files are sometimes harmful because they hide
>>> failures and web platform incompatibilities (e.g.
>>> ). So I think -expected.txt files for WPT
>>> tests should be rare and eventually be removed. An -expected.txt should not
>>> be treated as a part of the test itself because a) it doesn't exist in
>>> upstream WPT and 2) it doesn't describe the standard expected behavior,
>>> thus isn't necessarily placed besides the test.
>>>
>>> The directory name 'platform' may be misleading. 'baselines' may be a
>>> better name (but we should not rename until we decide what to do for
>>> generic baselines). For WPT tests, 'failures' is 

[blink-dev] Re: introduce blink_wpt_tests

['Weizhong Xia' via blink-dev]

Hi Folks

Another update on this work. Today I landed the CL to run blink_wpt_tests
as a different step on swarming bots. We will need to find test results at
their respective steps.

Feel free to ping me if you have any questions.

thanks, Weizhong


On Tue, May 3, 2022 at 9:03 AM Weizhong Xia  wrote:

> Hi Folks
>
> A follow up of this announcement. I was finally able to land this change
> yesterday, with a small change. We later decided to keep the platform, to
> reduce the size of the CL. So now all the baselines are under
> *web_tests/platform*, and generic baselines are under
> *web_tests/platform/generic*.
>
> If you have any ongoing CL that changes baselines, pls make sure the
> baselines are at the correct place. Feel free to ping me if you have any
> questions.
>
> thanks, Weizhong
>
> On Mon, Apr 25, 2022 at 4:06 PM Weizhong Xia  wrote:
>
>> Dear blink devs
>>
>> As some of you may already know, I am working to split blink_web_tests
>> into two test suites: *blink_web_tests* for legacy tests and
>> *blink_wpt_tests* for wpt tests(crbug/1299834 ).
>> We now have reached the first point to move all baselines to a central
>> place (CL
>> ).
>> Some of the changes you may need to be aware of:
>> 1. We renamed web_tests/platform to web_tests/baselines.
>> 2. All the generic baselines are now put under
>> web_tests/baselines/generic. (reference files are allowed at both
>> baselines/generic or in the same folder of the test itself.)
>>
>> I planned to do another round of rebaseline then land this CL the
>> coming Saturday, to reduce the chance of merge conflict.
>>
>> After that, we will have another CL to actually break blink_web_tests
>> into two. This should have no impact on your local experiences. The only
>> difference is that we are now running legacy tests and wpt tests at
>> different steps, so you need to get log/results at different places.
>>
>> At this point blink_wpt_tests will continue to run with run_web_tests.py.
>> We have a plan to switch it to use wptrunner, targeting the middle of next
>> year. After that we will have tools, web_tests, wpt_tests under blink, and
>> we will use metadata as test expectation for wpt_tests. More details to
>> come.
>>
>> Thanks for your attention. If you have any suggestion/comments, feel free
>> to reply here, or put that in the crbug or CL. I definitely want to get
>> more input from you folks.
>>
>> Thanks, Weizhong
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiqgL5jUtou9YBcvK8j_hzK-QtfEhFGEFrXM3XE6yY%2BnWg%40mail.gmail.com.

Re: [blink-dev] Developer pain as a result of "introducing blink_wpt_tests"

['Weizhong Xia' via blink-dev]

Hi Blink devs

FYI we are finally ready to move generic baselines back to their original
places: the test folders. I plan to land the CL today. To avoid any merge
conflict during this process, we will add an OWNERS file to
//third_party/blink/web_tests/platform/generic. Any change to that folder
will require an additional +1, and will not be approved. Once the move is
done, the "platform/generic" folder will be removed.

thanks, Weizhong

On Thu, Jun 23, 2022 at 11:00 AM Weizhong Xia  wrote:

> Hi blink devs
>
> Thanks to those who joined the survey at
> https://forms.gle/ju45qciS5VTR4ywN7. Most of you expressed the desire to
> put baselines at the same place of the tests. Your voice is heard, and here
> is the plan for the next step.
>
> In Q3 we will work on to completely separate legacy layout tests and wpt
> tests, to put them under `third_party/blink/web_tests` and
> `third_party/blink/wpt_tests` respectively. Generic baselines (including
> generic virtual baselines) will be moved back to their previous place.
> Rebaseline tool will be updated to work with this structure, and update
> baselines for legacy layout tests and wpt tests in a single run. We will
> have different copies of *TestExpectations*, *FlagSpecificConfig*,
> *VirtualTestSuites* etc for legacy tests and wpt tests. When working on
> those files, we will need to make sure we are updating the correct copy of
> the file. (We will investigate if we need some presubmit check for such a
> scenario).
>
> The reason for this is two fold: as requested we want to put baselines
> side to side to the tests, and we want to make the directory structure
> right to speed up the switch to wptrunner.
>
> Thoughts? Feel free to leave a comment in crbug/1299834
> .
>
> Thanks, Weizhong
>
>
>
> On Mon, Jun 6, 2022 at 5:00 PM Xianzhu Wang 
> wrote:
>
>> On Mon, Jun 6, 2022 at 3:52 PM Weizhong Xia  wrote:
>>
>>> Xianzhu, yes 'baselines' is the name we agreed on previously. The reason
>>> I later changed back to use 'platform' is because that will make the CL
>>> smaller, and make it easier for gerrit to handle it. We can make one round
>>> rename when everything is stabilized. (I left you a message when you are
>>> OOO. I am not sure if that message lived long enough for you to catch it.)
>>>
>>
>> I caught the message. I guessed that the name 'platform' might be one of
>> the reasons for the surprise to blink developers after the change, and the
>> name 'baselines' might make the change easier to explain :)
>>
>> On Mon, Jun 6, 2022 at 3:52 PM Domenic Denicola 
>> wrote:
>>
>>> I'm not sure I understand this logic. -expected.txt files are just
>>> more-convenient, more in-your-face versions of TestExpectations files.
>>> Surely you're not suggesting we get rid of TestExpectations files?
>>>
>>
>> Sorry, "-expected.txt ... should be ... eventually be removed" in my
>> previous email was not clear. I meant for each individual -expected.txt we
>> should eventually remove it because we should fix the failure. The same
>> logic applies to TestExpectations. At any time we may allow a certain
>> number of failures but we should keep the number as small as possible.
>>
>> I think we should prefer TestExpectations to -expected.txt for WPT tests
>> because the entries in TestExpectations have associated bugs which track
>> the fixing process, unless we find a better way to track the fixing of the
>> failures in -expected.txt. -expected.txt files do have their values, e.g.
>> for partially-passing tests we can discover regressions and progressions of
>> individual sub tests, but they should be rare.
>>
>> I think separating -expected.txt from the tests has the following
>> benefits:
>> - It makes it clear to blink developers that the files are not a part of
>> WPT.
>> - It simplifies the WPT export/import process and others by reducing
>> blink-specific files under external/wpt.
>>
>> It does make it more difficult to find -expected.txt, but we already have
>> the similar well-known logic for platform-specific baselines. Though
>> platform-specific baselines are rare, ignoring a platform baseline can
>> still cause surprises.
>>
>> I think we can improve the test result viewer
>> 
>> - to better show -expected.txt for passing tests
>> - to show information about tests without actually running the tests
>> WDYT?
>>
>>
>>>
>>> On Mon, Jun 6, 2022 at 6:30 PM Xianzhu Wang 
>>> wrote:
>>>
 I think we first need to answer a question: Why do we need
 *-expected.txt for WPT tests?

 Upstream WPT doesn't have *-expected.txt. *-expected.txt is a
 blink-specific thing to allow some failing WPT tests to pass on blink with
 temporarily allowed failures. If a test has -expected.txt, it means Chrome
 behaves differently than the standard web platform behavior (or the test
 itself is wrong). The files ar

Re: [blink-dev] Developer pain as a result of "introducing blink_wpt_tests"

['Weizhong Xia' via blink-dev]

I am glad we were able to make this happen. Thanks for all the feedback on
this particular issue and other blink/infra related issues. We definitely
need your feedback to make blink/infra better.

Thank you, all!

On Mon, Aug 29, 2022 at 9:07 AM Philip Jägenstedt 
wrote:

> I'd like to second that, thanks Weizhong!
>
> On Mon, Aug 29, 2022 at 5:07 AM Domenic Denicola 
> wrote:
>
>> This is great news! Thanks for being receptive to feedback and to working
>> to get us back to a more-productive state.
>>
>> On Sat, Aug 27, 2022 at 8:53 AM Xianzhu Wang 
>> wrote:
>>
>>> Hi Blink devs,
>>>
>>> The move has landed (https://crrev.com/1039971).
>>>
>>> If you have any pending CLs that add/remove/modify files under
>>> web_tests/platform/generic, please sync, and add/remove/modify the
>>> corresponding file under web_tests/ instead.
>>>
>>> Thanks,
>>> Xianzhu
>>>
>>> On Fri, Aug 26, 2022 at 10:57 AM Weizhong Xia 
>>> wrote:
>>>
 Hi Blink devs

 FYI we are finally ready to move generic baselines back to their
 original places: the test folders. I plan to land the CL today. To avoid
 any merge conflict during this process, we will add an OWNERS file to
 //third_party/blink/web_tests/platform/generic. Any change to that folder
 will require an additional +1, and will not be approved. Once the move is
 done, the "platform/generic" folder will be removed.

 thanks, Weizhong

 On Thu, Jun 23, 2022 at 11:00 AM Weizhong Xia 
 wrote:

> Hi blink devs
>
> Thanks to those who joined the survey at
> https://forms.gle/ju45qciS5VTR4ywN7. Most of you expressed the desire
> to put baselines at the same place of the tests. Your voice is heard, and
> here is the plan for the next step.
>
> In Q3 we will work on to completely separate legacy layout tests and
> wpt tests, to put them under `third_party/blink/web_tests` and
> `third_party/blink/wpt_tests` respectively. Generic baselines (including
> generic virtual baselines) will be moved back to their previous place.
> Rebaseline tool will be updated to work with this structure, and update
> baselines for legacy layout tests and wpt tests in a single run. We will
> have different copies of *TestExpectations*, *FlagSpecificConfig*,
> *VirtualTestSuites* etc for legacy tests and wpt tests. When working
> on those files, we will need to make sure we are updating the correct copy
> of the file. (We will investigate if we need some presubmit check for such
> a scenario).
>
> The reason for this is two fold: as requested we want to put baselines
> side to side to the tests, and we want to make the directory structure
> right to speed up the switch to wptrunner.
>
> Thoughts? Feel free to leave a comment in crbug/1299834
> .
>
> Thanks, Weizhong
>
>
>
> On Mon, Jun 6, 2022 at 5:00 PM Xianzhu Wang 
> wrote:
>
>> On Mon, Jun 6, 2022 at 3:52 PM Weizhong Xia 
>> wrote:
>>
>>> Xianzhu, yes 'baselines' is the name we agreed on previously. The
>>> reason I later changed back to use 'platform' is because that will make 
>>> the
>>> CL smaller, and make it easier for gerrit to handle it. We can make one
>>> round rename when everything is stabilized. (I left you a message when 
>>> you
>>> are OOO. I am not sure if that message lived long enough for you to 
>>> catch
>>> it.)
>>>
>>
>> I caught the message. I guessed that the name 'platform' might be one
>> of the reasons for the surprise to blink developers after the change, and
>> the name 'baselines' might make the change easier to explain :)
>>
>> On Mon, Jun 6, 2022 at 3:52 PM Domenic Denicola 
>> wrote:
>>
>>> I'm not sure I understand this logic. -expected.txt files are just
>>> more-convenient, more in-your-face versions of TestExpectations files.
>>> Surely you're not suggesting we get rid of TestExpectations files?
>>>
>>
>> Sorry, "-expected.txt ... should be ... eventually be removed" in my
>> previous email was not clear. I meant for each individual -expected.txt 
>> we
>> should eventually remove it because we should fix the failure. The same
>> logic applies to TestExpectations. At any time we may allow a certain
>> number of failures but we should keep the number as small as possible.
>>
>> I think we should prefer TestExpectations to -expected.txt for WPT
>> tests because the entries in TestExpectations have associated bugs which
>> track the fixing process, unless we find a better way to track the fixing
>> of the failures in -expected.txt. -expected.txt files do have their 
>> values,
>> e.g. for partially-passing tests we can discover regressions and
>> progressions of individual sub tests, but they should be rare.
>>
>> I think separating -expected.tx

[blink-dev] PSA: Virtual web test suits now have expiration dates

['Weizhong Xia' via blink-dev]

Hi Folks

TL;DR: We now support expiration dates for virtual test suites. All virtual
test suites now have an initial expiration date of July 1st, 2023. If you
own any virtual test suites, please review this value, and update that to
an appropriate date.

 In the past we have seen cases where virtual test suites are left running
when they should be deleted. By adding expiration dates, developers now
have a chance to review the test suites based on latest status. The
expiration date can be set to 'never' if necessary. An explanation is
required for such a case.

I will resent this a couple of times in case some people should miss this
due to the holiday season.

And happy holidays!

thanks, Weizhong

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiqsLPyk12-NRfbtZ5saj_TcQowmRz0AS4SDCUBe9uPRhg%40mail.gmail.com.

[blink-dev] PSA: Introduce collect_web_tests.py for web test

['Weizhong Xia' via blink-dev]

tl;dr: We introduced a mechanism that pre-collects test cases to
//third_party/blink/web_tests/AllTestsByDirectories.json, and uses a
presubmit check to ensure this file is up to date. You can stop here if you
don't write web test cases.

Dear blink-devs,

You may have noticed that when you try to upload a CL, you are prompted to
update AllTestsByDirectories.json if your CL changes web test cases. Yes,
today we landed a CL
 that
pre-collects test cases and saves that to the json file. The reason to do
this is that rwt tries to collect tests on every swarming bots, and that
takes about 30 seconds (the best case when running on SSD). This is not a
small amount of time as we are targeting a 10-min CQ. After this CL lands,
collecting tests now takes less than 2 seconds for all platforms.

Impact to you: if your CL adds/deletes/renames web test cases, you will be
prompted. Follow the instructions there should be enough. But if you still
have an issue, (even though we have tested many different scenarios we can
think about), feel free to ping/email me.

There is no change to how you run rwt locally. We added a new switch to
tell the swarming bots to load tests from the json file. This switch is
turned off by default. Developers usually don't need to turn this on,
unless all of your tests are saved to the json file and you are running a
full test so want to save about 30 seconds.

Thanks for your time!

Cheers, Weizhong

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiprRWy_UCC0w2oohbSGGTbeP%3DdkyYnBOx7j8Y-abDSe2A%40mail.gmail.com.

Re: [blink-dev] PSA: Introduce collect_web_tests.py for web test

['Weizhong Xia' via blink-dev]

Running collect_web_tests on external/wpt is kind of slow, because we need
to generate wpt manifest first. Running that on other folders will be fast.
And I will take a look to see if there is room for improvement.

One thing worth mentioning is that if the change in external/wpt is meant
to be exported to upstream <https://github.com/web-platform-tests/wpt>, you
can ignore the presubmit check warning. (Should have mentioned this in the
PSA).

On Tue, Aug 17, 2021 at 11:01 AM Dirk Pranke  wrote:

> On Tue, Aug 17, 2021 at 10:48 AM Ian Kilpatrick 
> wrote:
>
>> So I was a little surprised by this last night.
>>
>
> Yeah, we probably should've had the discussion first, sorry :(.
>
>
>>
>> 1. This was relatively slow on my macbook, running locally takes ~40s.
>> From just now:
>>
>> ikilpatrick-macbookpro:src ikilpatrick$ touch
>> third_party/blink/web_tests/external/wpt/css/css-tables/test.html
>>
>> ikilpatrick-macbookpro:src ikilpatrick$ time
>> third_party/blink/tools/collect_web_tests.py external/wpt
>>
>>
>> real 0m40.492s
>>
>> user 0m23.007s
>>
>> sys 0m41.715s
>>
>> 2. I had a bunch of uncommitted tests in various wpt directories that I
>> needed to move out to somewhere else (rather than the script just adding
>> committed tests).
>>
>
> One other concern is that we might end up with a lot of merge conflicts in
> a single file, and so we might want to shard by the top level directories
> or something ...
>
> (I'd also note that uploading patches is already relatively slow, do we
>> have metrics for how long developers are waiting for upload scripts to run?
>> - I suspect this is due to WPT presubmit scripts but unsure)
>>
>
> The upload checks are just generally crazy slow these days. I'm not sure
> if we have collected metrics for them, but things are an order of magnitude
> slower than I'd like them to be.
>
> But that's partially a separate topic (apart from this check, of course).
>
> -- Dirk
>
>
>> Ian
>>
>>
>>
>>
>> On Tue, Aug 17, 2021 at 10:20 AM 'Dirk Pranke' via blink-dev <
>> blink-dev@chromium.org> wrote:
>>
>>>
>>>
>>> On Tue, Aug 17, 2021 at 10:08 AM 'Weizhong Xia' via blink-dev <
>>> blink-dev@chromium.org> wrote:
>>>
>>>> tl;dr: We introduced a mechanism that pre-collects test cases to
>>>> //third_party/blink/web_tests/AllTestsByDirectories.json, and uses a
>>>> presubmit check to ensure this file is up to date. You can stop here if you
>>>> don't write web test cases.
>>>>
>>>> Dear blink-devs,
>>>>
>>>> You may have noticed that when you try to upload a CL, you are prompted
>>>> to update AllTestsByDirectories.json if your CL changes web test cases.
>>>> Yes, today we landed a CL
>>>> <https://chromium-review.googlesource.com/c/chromium/src/+/3067372>
>>>> that pre-collects test cases and saves that to the json file. The reason to
>>>> do this is that rwt tries to collect tests on every swarming bots, and that
>>>> takes about 30 seconds (the best case when running on SSD). This is not a
>>>> small amount of time as we are targeting a 10-min CQ. After this CL lands,
>>>> collecting tests now takes less than 2 seconds for all platforms.
>>>>
>>>> Impact to you: if your CL adds/deletes/renames web test cases, you will
>>>> be prompted. Follow the instructions there should be enough. But if you
>>>> still have an issue, (even though we have tested many different scenarios
>>>> we can think about), feel free to ping/email me.
>>>>
>>>> There is no change to how you run rwt locally. We added a new switch to
>>>> tell the swarming bots to load tests from the json file. This switch is
>>>> turned off by default. Developers usually don't need to turn this on,
>>>> unless all of your tests are saved to the json file and you are running a
>>>> full test so want to save about 30 seconds.
>>>>
>>>
>>> FWIW, I think this is an important and probably worthwhile change
>>> (disclaimer: I did review it and was involved in the design). The number of
>>> tests we have now is substantial and the startup delay is a significant
>>> cost. We need to be looking into ways to reduce this.
>>>
>>> I think it's possible we should turn this on by default and/or do some
>>> hooking to make things aware of when you

Re: [blink-dev] PSA: Introduce collect_web_tests.py for web test

['Weizhong Xia' via blink-dev]

FYI I just reverted the CL, and will take necessary actions once we reach
consensus.

Nate, yes if you plan to land your change in chromium, the presubmit check
will prevent the CL to land if the json file is not up to date. You should
be good now as the CL is reverted.

Thanks, Weizhong

On Tue, Aug 17, 2021 at 11:48 AM Nate Chapin  wrote:

> I hit this on
> https://chromium-review.googlesource.com/c/chromium/src/+/3098588
>
> The only test change is in external/wpt, and it's a presubmit error which
> will block the CQ, so I think as it stands currently, you can't ignore the
> warning even for external/wpt/.
>
> On Tue, Aug 17, 2021 at 11:08 AM 'Weizhong Xia' via blink-dev <
> blink-dev@chromium.org> wrote:
>
>> Running collect_web_tests on external/wpt is kind of slow, because we
>> need to generate wpt manifest first. Running that on other folders will be
>> fast. And I will take a look to see if there is room for improvement.
>>
>> One thing worth mentioning is that if the change in external/wpt is meant
>> to be exported to upstream <https://github.com/web-platform-tests/wpt>,
>> you can ignore the presubmit check warning. (Should have mentioned this in
>> the PSA).
>>
>> On Tue, Aug 17, 2021 at 11:01 AM Dirk Pranke  wrote:
>>
>>> On Tue, Aug 17, 2021 at 10:48 AM Ian Kilpatrick <
>>> ikilpatr...@chromium.org> wrote:
>>>
>>>> So I was a little surprised by this last night.
>>>>
>>>
>>> Yeah, we probably should've had the discussion first, sorry :(.
>>>
>>>
>>>>
>>>> 1. This was relatively slow on my macbook, running locally takes ~40s.
>>>> From just now:
>>>>
>>>> ikilpatrick-macbookpro:src ikilpatrick$ touch
>>>> third_party/blink/web_tests/external/wpt/css/css-tables/test.html
>>>>
>>>> ikilpatrick-macbookpro:src ikilpatrick$ time
>>>> third_party/blink/tools/collect_web_tests.py external/wpt
>>>>
>>>>
>>>> real 0m40.492s
>>>>
>>>> user 0m23.007s
>>>>
>>>> sys 0m41.715s
>>>>
>>>> 2. I had a bunch of uncommitted tests in various wpt directories that I
>>>> needed to move out to somewhere else (rather than the script just adding
>>>> committed tests).
>>>>
>>>
>>> One other concern is that we might end up with a lot of merge conflicts
>>> in a single file, and so we might want to shard by the top level
>>> directories or something ...
>>>
>>> (I'd also note that uploading patches is already relatively slow, do we
>>>> have metrics for how long developers are waiting for upload scripts to run?
>>>> - I suspect this is due to WPT presubmit scripts but unsure)
>>>>
>>>
>>> The upload checks are just generally crazy slow these days. I'm not sure
>>> if we have collected metrics for them, but things are an order of magnitude
>>> slower than I'd like them to be.
>>>
>>> But that's partially a separate topic (apart from this check, of course).
>>>
>>> -- Dirk
>>>
>>>
>>>> Ian
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 17, 2021 at 10:20 AM 'Dirk Pranke' via blink-dev <
>>>> blink-dev@chromium.org> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 17, 2021 at 10:08 AM 'Weizhong Xia' via blink-dev <
>>>>> blink-dev@chromium.org> wrote:
>>>>>
>>>>>> tl;dr: We introduced a mechanism that pre-collects test cases to
>>>>>> //third_party/blink/web_tests/AllTestsByDirectories.json, and uses a
>>>>>> presubmit check to ensure this file is up to date. You can stop here if 
>>>>>> you
>>>>>> don't write web test cases.
>>>>>>
>>>>>> Dear blink-devs,
>>>>>>
>>>>>> You may have noticed that when you try to upload a CL, you are
>>>>>> prompted to update AllTestsByDirectories.json if your CL changes web test
>>>>>> cases. Yes, today we landed a CL
>>>>>> <https://chromium-review.googlesource.com/c/chromium/src/+/3067372>
>>>>>> that pre-collects test cases and saves that to the json file. The reason 
>>>>>> to
>>>>>> do this is that rwt tries to collect tests on every swarming bots, and 
>&g

[blink-dev] Slow tests that timeouts

['Weizhong Xia' via blink-dev]

Dear blink-devs,

As another effort trying to speed up blink_web_tests on CQ, We are looking
at slow tests that also timeouts. From the timing stats(example
)
and some research, there are about 0.25% such tests, but running such tests
takes about 4% of the total run time. So we are thinking of
doing something about it.

We would like to propose as below, and get feedback on this:
1. We add Skip to test expectations for slow tests that also timeouts. This
will effectively skip all such tests on swarming bots, and save ~50s real
time on linux-rel.
2. We add a presubmit check to make any unskipped Slow+Timeout an error.
(We can also make this a warning, but such warning will persist, so kind of
annoying)
3. Devs are encouraged to take actions to either fix the timeout, or break
the slow tests into pieces to make it not slow.

Please let us know your opinions on this.

Thanks, Weizhong

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipU3NDSp7tWd%3DqA%3DrZWMrmXjfT4NzTG7U8EoEAgsQ1QXw%40mail.gmail.com.

Re: [blink-dev] Slow tests that timeouts

['Weizhong Xia' via blink-dev]

>> I recently found a test that was TIMEOUT in TestExpectations, and still
timed out in Slow, so I left it in TestExpectations under the assumption
that it will timeout faster there.
What do you mean "timed out in Slow"? A test did not mark as slow, but
timeout after 60 seconds? Worth mentioning some slow wpt tests are given
the same timeout value as slow tests in the code.

But I did see some tests timed out at some abnormal value. For example
tests below timeouts at 22 second. Not sure how this could happen.
19:13:50.493 20120 [3522/4178]
external/wpt/webrtc-identity/idlharness.https.window.html
failed unexpectedly (test timed out) 22.3300s

On Wed, Aug 18, 2021 at 11:26 AM Stephen Chenney 
wrote:

>
>
> On Wed, Aug 18, 2021 at 12:01 PM 'Weizhong Xia' via blink-dev <
> blink-dev@chromium.org> wrote:
>
>> Dear blink-devs,
>>
>> As another effort trying to speed up blink_web_tests on CQ, We are
>> looking at slow tests that also timeouts. From the timing stats(example
>> <https://test-results.appspot.com/data/layout_results/linux-rel/773524/blink_web_tests%20%28with%20patch%29/layout-test-results/results.html>)
>> and some research, there are about 0.25% such tests, but running such tests
>> takes about 4% of the total run time. So we are thinking of
>> doing something about it.
>>
>> We would like to propose as below, and get feedback on this:
>> 1. We add Skip to test expectations for slow tests that also timeouts.
>> This will effectively skip all such tests on swarming bots, and save ~50s
>> real time on linux-rel.
>> 2. We add a presubmit check to make any unskipped Slow+Timeout an error.
>> (We can also make this a warning, but such warning will persist, so kind of
>> annoying)
>> 3. Devs are encouraged to take actions to either fix the timeout, or
>> break the slow tests into pieces to make it not slow.
>>
>>
> I recently found a test that was TIMEOUT in TestExpectations, and still
> timed out in Slow, so I left it in TestExpectations under the assumption
> that it will timeout faster there.
>
> The underlying issue is not a slow test but a test that hangs and we may
> or may not want to skip such a test. Would identifying these cases and
> moving them from Slow to TestExpectations have any noticeable impact?
>
>
>> Please let us know your opinions on this.
>>
>> Thanks, Weizhong
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "blink-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to blink-dev+unsubscr...@chromium.org.
>> To view this discussion on the web visit
>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipU3NDSp7tWd%3DqA%3DrZWMrmXjfT4NzTG7U8EoEAgsQ1QXw%40mail.gmail.com
>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipU3NDSp7tWd%3DqA%3DrZWMrmXjfT4NzTG7U8EoEAgsQ1QXw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSip%3DRz%3D46ZfZ5nyg10ORBwKg2tcFAX%2Bh0di%3D98Q50R5%3DrA%40mail.gmail.com.

[blink-dev] PSA: explicitly listing data dependency for blink_web_tests

['Weizhong Xia' via blink-dev]

Dear blink devs

As of now blink_web_tests has a data dependency on
//third_party/blink/web_tests. I made a CL
 that
listed the next level directories as the data dependency for
blink_web_tests. By doing this, we can skip some baselines for some
platforms. For example, we never use linux baselines on Windows.

Even though the sub directories under web_tests are pretty stable nowadays,
we added a presubmit check to make sure all the test directories are listed
in //BUILD.gn.  If it happens you need to add a new directory under
web_tests, please also update //BUILD.gn to list the new folder under the
mark "# === List Test Cases folders here ===". Otherwise, the presubmit
check will give a warning for that.

In this CL we also assumed flag-specific tests only run on linux-bots. Pls
speak up if you know this is not the case. A note is added to
//docs/testing/web_tests.md, and if you need to run flag-specific tests on
some other platforms in future, please remember to update //BUILD.gn also.

thanks, Weizhong

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirYfHLj%2BKcv-KCaAR5%3DMpPbNMXraNc7xZ6S45KzCEujQQ%40mail.gmail.com.

Re: [blink-dev] PSA: explicitly listing data dependency for blink_web_tests

['Weizhong Xia' via blink-dev]

I did search the source code before I made the change. Seems the os type is
always ubuntu for flag-specific builders. And CQ dry run can actually pass.
I would imagine the test will fail if we are missing so many test
expectations.

Virtual tests are different to flag-specific tests, right?

On Tue, Sep 7, 2021 at 9:59 AM Stefan Zager  wrote:

> On Tue, Sep 7, 2021 at 9:01 AM 'Weizhong Xia' via blink-dev <
> blink-dev@chromium.org> wrote:
>
>>
>> In this CL we also assumed flag-specific tests only run on linux-bots.
>> Pls speak up if you know this is not the case.
>>
>
> I don't think that's true. Virtual test suites run on all platforms, and
> they rely on expectation files in flag-specific/.
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipo0XRY6DD7%2B6ivE3LsbmmPsfb9DBwZm1eN_7yXSC00fg%40mail.gmail.com.

Re: [blink-dev] Fwd: Can't run Web Tests locally on MacOS 12 device

['Weizhong Xia' via blink-dev]

I have two CLs trying to create the 3pp package for this. Both of them met
some kind of compile issue. I am in discussion with ppl to see how to fix
that.

As mentioned in that crbug, we are able to successfully build httpd for mac
arm with the checked-in script. I am not sure if this makes a viable
workaround for those who need it urgently.

I have been getting different signals about the priority.

thanks, Weizhong

On Thu, Dec 16, 2021 at 3:19 PM Nico Weber  wrote:

> Hello,
>
> Yes, that is known:
> https://bugs.chromium.org/p/chromium/issues/detail?id=1190885#c57 You can
> still run tests on macOS 11.
>
> However, the old version doesn't work on arm machines. So
> reverting doesn't work either.
>
> So we need to try to quickly finish the work to make a 3pp httpd package.
>
>
> On Thu, Dec 16, 2021 at 5:32 PM Dave Tapuska 
> wrote:
>
>> FYI... Seems PHP was removed in macOS Monterey. Nico sounds like we need
>> to revert your patch.
>>
>> dave
>>
>> On Thu, Dec 16, 2021 at 5:20 PM Dave Tapuska 
>> wrote:
>>
>>> I get it too.. I wonder if
>>> https://source.chromium.org/chromium/chromium/src/+/1156f94e35e73182126f17cf52c45f70a8d4c45b
>>> is the cause because my default system http conf doesn't have a PHP library.
>>>
>>> Nico?
>>>
>>> dave.
>>>
>>> On Thu, Dec 16, 2021 at 5:01 PM 'Alexis Hétu' via blink-dev <
>>> blink-dev@chromium.org> wrote:
>>>
 +blink-dev

 -- Forwarded message -
 De : Alexis Hétu 
 Date: jeu. 16 déc. 2021, à 16 h 42
 Subject: Can't run Web Tests locally on MacOS 12 device
 To: 


 Hi all,

   When I try to run Web Tests locally on my 2017 iMac Pro or 2018
 Macbook Pro, both on MacOS 12, I get this error:

 No code signing authority for module at /usr/libexec/apache2/libphp7.so
 specified in LoadModule directive
   How do I fix that?

 Thanks

 Alexis

 --
 You received this message because you are subscribed to the Google
 Groups "blink-dev" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to blink-dev+unsubscr...@chromium.org.
 To view this discussion on the web visit
 https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPB1aKAp9Nz9Dk%2BFQ5xXL0BmUeYkPAE%2Bjfs54OiRd2JA3wBc7Q%40mail.gmail.com
 
 .

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSio%2BUNojerGmZkHUvJr7EtcNAcYBouEgV%2BKmgabUGuXDmA%40mail.gmail.com.

Re: [blink-dev] Fwd: Can't run Web Tests locally on MacOS 12 device

['Weizhong Xia' via blink-dev]

Folks

I made a CL
 that
disables php support in httpd for Mac12. With this change we will be able
to run blink_web_tests locally on Mac12, with the side effect that tests
that require php will fail. This is a temporary workaround to unblock those
who need to run the test locally on mac12. This CL will be reverted once
the work for 3pp packaging httpd is done, and another email will be sent
out.

There is no impact to other platforms: win, linux or other mac versions.

thanks, Weizhong

On Thu, Dec 16, 2021 at 4:57 PM Weizhong Xia  wrote:

> I have two CLs trying to create the 3pp package for this. Both of them met
> some kind of compile issue. I am in discussion with ppl to see how to fix
> that.
>
> As mentioned in that crbug, we are able to successfully build httpd for
> mac arm with the checked-in script. I am not sure if this makes a viable
> workaround for those who need it urgently.
>
> I have been getting different signals about the priority.
>
> thanks, Weizhong
>
> On Thu, Dec 16, 2021 at 3:19 PM Nico Weber  wrote:
>
>> Hello,
>>
>> Yes, that is known:
>> https://bugs.chromium.org/p/chromium/issues/detail?id=1190885#c57 You
>> can still run tests on macOS 11.
>>
>> However, the old version doesn't work on arm machines. So
>> reverting doesn't work either.
>>
>> So we need to try to quickly finish the work to make a 3pp httpd package.
>>
>>
>> On Thu, Dec 16, 2021 at 5:32 PM Dave Tapuska 
>> wrote:
>>
>>> FYI... Seems PHP was removed in macOS Monterey. Nico sounds like we need
>>> to revert your patch.
>>>
>>> dave
>>>
>>> On Thu, Dec 16, 2021 at 5:20 PM Dave Tapuska 
>>> wrote:
>>>
 I get it too.. I wonder if
 https://source.chromium.org/chromium/chromium/src/+/1156f94e35e73182126f17cf52c45f70a8d4c45b
 is the cause because my default system http conf doesn't have a PHP 
 library.

 Nico?

 dave.

 On Thu, Dec 16, 2021 at 5:01 PM 'Alexis Hétu' via blink-dev <
 blink-dev@chromium.org> wrote:

> +blink-dev
>
> -- Forwarded message -
> De : Alexis Hétu 
> Date: jeu. 16 déc. 2021, à 16 h 42
> Subject: Can't run Web Tests locally on MacOS 12 device
> To: 
>
>
> Hi all,
>
>   When I try to run Web Tests locally on my 2017 iMac Pro or 2018
> Macbook Pro, both on MacOS 12, I get this error:
>
> No code signing authority for module at
> /usr/libexec/apache2/libphp7.so specified in LoadModule directive
>   How do I fix that?
>
> Thanks
>
> Alexis
>
> --
> You received this message because you are subscribed to the Google
> Groups "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to blink-dev+unsubscr...@chromium.org.
> To view this discussion on the web visit
> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPB1aKAp9Nz9Dk%2BFQ5xXL0BmUeYkPAE%2Bjfs54OiRd2JA3wBc7Q%40mail.gmail.com
> 
> .
>


-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipwisD6Fie2j3XnzFcpEzXTbtjJmHZ_uDtqf8Zji3XD3g%40mail.gmail.com.

Re: [blink-dev] Fwd: Can't run Web Tests locally on MacOS 12 device

['Weizhong Xia' via blink-dev]

Hi Jan

Thanks for reporting this. But I just run this on my Mac (version 12.2),
and did not see this issue. Can you pls file a crbug against me and include
all the details?

thanks, Weizhong

On Mon, Feb 7, 2022 at 3:20 AM Jan Keitel  wrote:

> Weizhong,
>
> Thank you for the patch - very helpful on MacOS 12.1. One naive question:
> Unless I comment out
> https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/tools/blinkpy/common/system/platform_info.py;l=195
> the assert still stops me from running the web tests on my Mac OS 12.1
> machine. Is there a flag that I can set to avoid having to do this or
> should this also be included in a CL?
>
> Thanks,
> Jan
>
> On Thursday, January 6, 2022 at 9:41:31 PM UTC+1 Alexis Hétu wrote:
>
>> Thanks Weizhong! This will help local debugging of Web Tests on MacOS
>> corp devices.
>>
>> Le jeu. 6 janv. 2022, à 15 h 36, Weizhong Xia  a
>> écrit :
>>
>>> Folks
>>>
>>> I made a CL
>>> 
>>> that disables php support in httpd for Mac12. With this change we will be
>>> able to run blink_web_tests locally on Mac12, with the side effect that
>>> tests that require php will fail. This is a temporary workaround to unblock
>>> those who need to run the test locally on mac12. This CL will be reverted
>>> once the work for 3pp packaging httpd is done, and another email will be
>>> sent out.
>>>
>>> There is no impact to other platforms: win, linux or other mac versions.
>>>
>>> thanks, Weizhong
>>>
>>> On Thu, Dec 16, 2021 at 4:57 PM Weizhong Xia  wrote:
>>>
 I have two CLs trying to create the 3pp package for this. Both of them
 met some kind of compile issue. I am in discussion with ppl to see how to
 fix that.

 As mentioned in that crbug, we are able to successfully build httpd for
 mac arm with the checked-in script. I am not sure if this makes a viable
 workaround for those who need it urgently.

 I have been getting different signals about the priority.

 thanks, Weizhong

 On Thu, Dec 16, 2021 at 3:19 PM Nico Weber  wrote:

> Hello,
>
> Yes, that is known:
> https://bugs.chromium.org/p/chromium/issues/detail?id=1190885#c57 You
> can still run tests on macOS 11.
>
> However, the old version doesn't work on arm machines. So
> reverting doesn't work either.
>
> So we need to try to quickly finish the work to make a 3pp httpd
> package.
>
>
> On Thu, Dec 16, 2021 at 5:32 PM Dave Tapuska 
> wrote:
>
>> FYI... Seems PHP was removed in macOS Monterey. Nico sounds like we
>> need to revert your patch.
>>
>> dave
>>
>> On Thu, Dec 16, 2021 at 5:20 PM Dave Tapuska 
>> wrote:
>>
>>> I get it too.. I wonder if
>>> https://source.chromium.org/chromium/chromium/src/+/1156f94e35e73182126f17cf52c45f70a8d4c45b
>>> is the cause because my default system http conf doesn't have a PHP 
>>> library.
>>>
>>> Nico?
>>>
>>> dave.
>>>
>>> On Thu, Dec 16, 2021 at 5:01 PM 'Alexis Hétu' via blink-dev <
>>> blin...@chromium.org> wrote:
>>>
 +blink-dev

 -- Forwarded message -
 De : Alexis Hétu 
 Date: jeu. 16 déc. 2021, à 16 h 42
 Subject: Can't run Web Tests locally on MacOS 12 device
 To: 


 Hi all,

   When I try to run Web Tests locally on my 2017 iMac Pro or 2018
 Macbook Pro, both on MacOS 12, I get this error:

 No code signing authority for module at
 /usr/libexec/apache2/libphp7.so specified in LoadModule directive
   How do I fix that?

 Thanks

 Alexis

 --
 You received this message because you are subscribed to the Google
 Groups "blink-dev" group.
 To unsubscribe from this group and stop receiving emails from it,
 send an email to blink-dev+...@chromium.org.
 To view this discussion on the web visit
 https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPB1aKAp9Nz9Dk%2BFQ5xXL0BmUeYkPAE%2Bjfs54OiRd2JA3wBc7Q%40mail.gmail.com
 
 .

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiqmz8mNK1NoiLAqxeetoDvG48ecg%2B42cVPUTywZp%3DOViA%40mail.gmail.com.

Re: [blink-dev] Fwd: Can't run Web Tests locally on MacOS 12 device

['Weizhong Xia' via blink-dev]

Hi Folks

FYI we have created the cipd package for mac, and blink_web_tests is
switched to use httpd from that package. The previous workaround had been
reverted. To run run_web_tests.py locally, pls run "gclient sync" to
install the package first.

thanks, Weizhong

On Mon, Feb 7, 2022 at 11:35 AM Jan Keitel  wrote:

> Hi Weizhong,
>
> sure, here you go:
> https://bugs.chromium.org/p/chromium/issues/detail?id=1294954
>
> Cheers,
> Jan
>
> On Monday, February 7, 2022 at 7:52:01 PM UTC+1 weiz...@google.com wrote:
>
>> Hi Jan
>>
>> Thanks for reporting this. But I just run this on my Mac (version 12.2),
>> and did not see this issue. Can you pls file a crbug against me and include
>> all the details?
>>
>> thanks, Weizhong
>>
>> On Mon, Feb 7, 2022 at 3:20 AM Jan Keitel  wrote:
>>
>>> Weizhong,
>>>
>>> Thank you for the patch - very helpful on MacOS 12.1. One naive
>>> question: Unless I comment out
>>> https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/tools/blinkpy/common/system/platform_info.py;l=195
>>> the assert still stops me from running the web tests on my Mac OS 12.1
>>> machine. Is there a flag that I can set to avoid having to do this or
>>> should this also be included in a CL?
>>>
>>> Thanks,
>>> Jan
>>>
>>> On Thursday, January 6, 2022 at 9:41:31 PM UTC+1 Alexis Hétu wrote:
>>>
 Thanks Weizhong! This will help local debugging of Web Tests on MacOS
 corp devices.

 Le jeu. 6 janv. 2022, à 15 h 36, Weizhong Xia  a
 écrit :

> Folks
>
> I made a CL
> 
> that disables php support in httpd for Mac12. With this change we will be
> able to run blink_web_tests locally on Mac12, with the side effect that
> tests that require php will fail. This is a temporary workaround to 
> unblock
> those who need to run the test locally on mac12. This CL will be reverted
> once the work for 3pp packaging httpd is done, and another email will be
> sent out.
>
> There is no impact to other platforms: win, linux or other mac
> versions.
>
> thanks, Weizhong
>
> On Thu, Dec 16, 2021 at 4:57 PM Weizhong Xia 
> wrote:
>
>> I have two CLs trying to create the 3pp package for this. Both of
>> them met some kind of compile issue. I am in discussion with ppl to see 
>> how
>> to fix that.
>>
>> As mentioned in that crbug, we are able to successfully build httpd
>> for mac arm with the checked-in script. I am not sure if this makes a
>> viable workaround for those who need it urgently.
>>
>> I have been getting different signals about the priority.
>>
>> thanks, Weizhong
>>
>> On Thu, Dec 16, 2021 at 3:19 PM Nico Weber 
>> wrote:
>>
>>> Hello,
>>>
>>> Yes, that is known:
>>> https://bugs.chromium.org/p/chromium/issues/detail?id=1190885#c57
>>> You can still run tests on macOS 11.
>>>
>>> However, the old version doesn't work on arm machines. So
>>> reverting doesn't work either.
>>>
>>> So we need to try to quickly finish the work to make a 3pp httpd
>>> package.
>>>
>>>
>>> On Thu, Dec 16, 2021 at 5:32 PM Dave Tapuska 
>>> wrote:
>>>
 FYI... Seems PHP was removed in macOS Monterey. Nico sounds like we
 need to revert your patch.

 dave

 On Thu, Dec 16, 2021 at 5:20 PM Dave Tapuska 
 wrote:

> I get it too.. I wonder if
> https://source.chromium.org/chromium/chromium/src/+/1156f94e35e73182126f17cf52c45f70a8d4c45b
> is the cause because my default system http conf doesn't have a PHP 
> library.
>
> Nico?
>
> dave.
>
> On Thu, Dec 16, 2021 at 5:01 PM 'Alexis Hétu' via blink-dev <
> blin...@chromium.org> wrote:
>
>> +blink-dev
>>
>> -- Forwarded message -
>> De : Alexis Hétu 
>> Date: jeu. 16 déc. 2021, à 16 h 42
>> Subject: Can't run Web Tests locally on MacOS 12 device
>> To: 
>>
>>
>> Hi all,
>>
>>   When I try to run Web Tests locally on my 2017 iMac Pro or 2018
>> Macbook Pro, both on MacOS 12, I get this error:
>>
>> No code signing authority for module at
>> /usr/libexec/apache2/libphp7.so specified in LoadModule directive
>>   How do I fix that?
>>
>> Thanks
>>
>> Alexis
>>
>> --
>> You received this message because you are subscribed to the
>> Google Groups "blink-dev" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to blink-dev+...@chromium.org.
>> To view this discussion on the web visit
>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPB1aK

Re: [blink-dev] Re: How to get notified when WPT auto-roller disables a test?

['Weizhong Xia' via blink-dev]

An update to this: we have made this CL 
 that 
implement the ideas proposed above. All the folders are defaultly opt-in 
unless the owners choose to opt out. The bugs will be filed against 
blink>infra>ecosystem if no other components have been defined in the 
directory tree. You are encouraged to set the right component for your 
tests so that the importer knows where to file crbugs and we don't have to 
reassign a crbug later.

Thanks, Weizhong

On Monday, May 21, 2018 at 5:08:12 AM UTC-7 Philip Jägenstedt wrote:

> On Mon, May 21, 2018 at 2:06 PM Stephen Mcgruer  
> wrote:
>
>> > it would be even less discoverable than the current opt-in
>>
>> If you're going to file bugs automatically (woo! totally support this), 
>> then why not just put a brief paragraph about WPT-NOTIFY: false at the end? 
>> E.g.
>>
>> "This bug was filed automatically due to a new WPT test failure for which 
>> you are marked an OWNER. If you don't want to receive these reports, please 
>> add WPT-NOTIFY: false to the relevant OWNERs file (
>> bit.ly/wpt-notify-false)."
>>
>
> Good suggestion, that would do it! Robert, can you file a bug about these 
> suggested changes for people to follow?
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/821577bf-82be-431f-b123-cb88e6101dd7n%40chromium.org.

[blink-dev] (Resending) Help needed: piloting run_wpt_tests.py and update metadata tool

['Weizhong Xia' via blink-dev]

Hey Folks

We are about to launch Wptrunner in chromium. For those working on WPT, can
you pls take some time to try out the tools and report any issues you
discovered? If you are interested, pls register through the link below.

https://docs.google.com/forms/d/17PPFPA_qz4L-0BOiyeCgrulAwMg-MvXU40hOvN9bx6o/edit?resourcekey=0-wsymaUfH1dW08X2GpIrl9g

thanks, Weizhong

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirNrL57nZjR-rZjamRJ0YwDLUHVshf_mTQRieK7LOOxeg%40mail.gmail.com.

[blink-dev] PSA: New baseline format for test-harness tests

['Weizhong Xia' via blink-dev]

Hi Blink Developers

We want to let you know that we are in the process of converting baselines
for test-harness tests into a new format. The reason is that we want to
make baselines machine-parsable so that we can regenerate Wpt Metadata from
Test Expectations and Baselines.

We are doing this directory by directory. Until now we have done this for
all legacy web tests, wpt_internal and external/wpt/html, and we are
expecting to complete this work by the end of this week.

This should mostly have no impact on your work, but a rebaseline might be
needed if we happened to work on the same directory at the same time.

thanks, Weizhong

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiobktBNXd1WD%2BK9gnaqAgW-oAq5Fn_3xGG1H9fUZ8Ezig%40mail.gmail.com.

Re: [blink-dev] PSA: New baseline format for test-harness tests

['Weizhong Xia' via blink-dev]

Yes, here is the design document:
https://docs.google.com/document/d/1FsbRm2gXwFnhyJYaE-wz2fagRABkuHCK4KCYb6JEhMQ/edit?resourcekey=0-47lxaiexsP8wfuzOtQhkXw#heading=h.103wg95souql

thanks, Weizhong

On Tue, Oct 17, 2023 at 10:54 AM Christian Biesinger <
cbiesin...@chromium.org> wrote:

> Interesting! When you say "machine parsable", I assume you mean you want
> to detect which subtests passed/failed? Could you share an example of
> the new format? Thanks!
>
> On Mon, Oct 16, 2023 at 9:07 PM 'Weizhong Xia' via blink-dev <
> blink-dev@chromium.org> wrote:
>
>> Hi Blink Developers
>>
>> We want to let you know that we are in the process of converting
>> baselines for test-harness tests into a new format. The reason is that we
>> want to make baselines machine-parsable so that we can regenerate Wpt
>> Metadata from Test Expectations and Baselines.
>>
>> We are doing this directory by directory. Until now we have done this for
>> all legacy web tests, wpt_internal and external/wpt/html, and we are
>> expecting to complete this work by the end of this week.
>>
>> This should mostly have no impact on your work, but a rebaseline might be
>> needed if we happened to work on the same directory at the same time.
>>
>> thanks, Weizhong
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "blink-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to blink-dev+unsubscr...@chromium.org.
>> To view this discussion on the web visit
>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiobktBNXd1WD%2BK9gnaqAgW-oAq5Fn_3xGG1H9fUZ8Ezig%40mail.gmail.com
>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiobktBNXd1WD%2BK9gnaqAgW-oAq5Fn_3xGG1H9fUZ8Ezig%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiqmvkh7Z%3DC9ZaT1QJLaceCSZOk-7DrYkQaWA3FEwVk2wQ%40mail.gmail.com.

Re: [blink-dev] PSA: New baseline format for test-harness tests

['Weizhong Xia' via blink-dev]

Folks, FYI we were finally able to make changes to remove the PASS lines in
baselines, which many of you have desired. With that the work to transit to
the new baseline format is all done.

https://chromium-review.googlesource.com/c/chromium/src/+/5046574
https://chromium-review.googlesource.com/c/chromium/src/+/5047251

thanks, Weizhong

On Tue, Oct 17, 2023 at 11:49 AM Christian Biesinger <
cbiesin...@chromium.org> wrote:

> Thanks, this looks great. I particularly like the omission of PASS lines.
>
> Christian
>
> On Tue, Oct 17, 2023 at 2:33 PM Weizhong Xia  wrote:
>
>> Yes, here is the design document:
>> https://docs.google.com/document/d/1FsbRm2gXwFnhyJYaE-wz2fagRABkuHCK4KCYb6JEhMQ/edit?resourcekey=0-47lxaiexsP8wfuzOtQhkXw#heading=h.103wg95souql
>>
>> thanks, Weizhong
>>
>> On Tue, Oct 17, 2023 at 10:54 AM Christian Biesinger <
>> cbiesin...@chromium.org> wrote:
>>
>>> Interesting! When you say "machine parsable", I assume you mean you want
>>> to detect which subtests passed/failed? Could you share an example of
>>> the new format? Thanks!
>>>
>>> On Mon, Oct 16, 2023 at 9:07 PM 'Weizhong Xia' via blink-dev <
>>> blink-dev@chromium.org> wrote:
>>>
>>>> Hi Blink Developers
>>>>
>>>> We want to let you know that we are in the process of converting
>>>> baselines for test-harness tests into a new format. The reason is that we
>>>> want to make baselines machine-parsable so that we can regenerate Wpt
>>>> Metadata from Test Expectations and Baselines.
>>>>
>>>> We are doing this directory by directory. Until now we have done this
>>>> for all legacy web tests, wpt_internal and external/wpt/html, and we are
>>>> expecting to complete this work by the end of this week.
>>>>
>>>> This should mostly have no impact on your work, but a rebaseline might
>>>> be needed if we happened to work on the same directory at the same time.
>>>>
>>>> thanks, Weizhong
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "blink-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to blink-dev+unsubscr...@chromium.org.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiobktBNXd1WD%2BK9gnaqAgW-oAq5Fn_3xGG1H9fUZ8Ezig%40mail.gmail.com
>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiobktBNXd1WD%2BK9gnaqAgW-oAq5Fn_3xGG1H9fUZ8Ezig%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipM0Dz1OzX3YT89JN96BAYMbPfZyvaV-x7fbVqdn-YXSw%40mail.gmail.com.

[blink-dev] PSA: chrome_wpt_tests has been added to CI/CQ

['Weizhong Xia' via blink-dev]

Hello Fellow Developers!

This email is a PSA for running Web platform tests against Chrome on CI/CQ.

What?


   -

   A new test suite “chrome_wpt_tests” has been added to the CI/CQ, which
   will run web platform tests against Chrome.


Why?


   -

   Data has shown about 2% of WPTs have different test results when running
   against Chrome v.s. content_shell.
   -

   Running WPTs against Chrome to prevent Chrome-specific regressions.


When?


   -

   Dec 05, 2023


Need help?


   -

   See our documentation
   

   for run_wpt_tests.py
   -

   File a bug against blink>infra


Thanks,
Weizhong on behalf of Blink Engprod

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiqBhtJZGjO4_6OJJh1tkgprCLd6pw0YyS8_3DeC72XGAA%40mail.gmail.com.

[blink-dev] Re: PSA: chrome_wpt_tests has been added to CI/CQ

['Weizhong Xia' via blink-dev]

Dear Fellow developers,

Some of you have reported that the new chrome_wpt_tests

suite is causing problems for you as it is testing against the old headless
mode Chrome. Due to that we have decided to change the test suite to
experimental mode for now(CL
). We
will evaluate what would be the best next step to re-enable the test suite
and update here once we have the plan locked down.

Sorry for the inconvenience.

- Weizhong


On Tue, Dec 5, 2023 at 11:29 AM Weizhong Xia  wrote:

> Hello Fellow Developers!
>
> This email is a PSA for running Web platform tests against Chrome on CI/CQ.
>
> What?
>
>
>-
>
>A new test suite “chrome_wpt_tests” has been added to the CI/CQ, which
>will run web platform tests against Chrome.
>
>
> Why?
>
>
>-
>
>Data has shown about 2% of WPTs have different test results when
>running against Chrome v.s. content_shell.
>-
>
>Running WPTs against Chrome to prevent Chrome-specific regressions.
>
>
> When?
>
>
>-
>
>Dec 05, 2023
>
>
> Need help?
>
>
>-
>
>See our documentation
>
> 
>for run_wpt_tests.py
>-
>
>File a bug against blink>infra
>
>
> Thanks,
> Weizhong on behalf of Blink Engprod
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSioBdHsExE-4PX_XJ4ZYw3TFdie%2B8Cdw8wn3hNRPNtUHXQ%40mail.gmail.com.

[blink-dev] Re: PSA: Virtual web test suits now have expiration dates

['Weizhong Xia' via blink-dev]

Hi Folks

This is a reminder as we are approaching July 1st, 2023, the initial
expiration date I set for all the virtual test suites. I did a search just
now. Most VTS would still expire at that date (161 out of 194).

Can we review this to make sure we do not lose coverage suddenly?

thanks, Weizhong

On Tue, Nov 22, 2022 at 12:07 PM Weizhong Xia  wrote:

> Hi Folks
>
> TL;DR: We now support expiration dates for virtual test suites. All
> virtual test suites now have an initial expiration date of July 1st, 2023.
> If you own any virtual test suites, please review this value, and update
> that to an appropriate date.
>
>  In the past we have seen cases where virtual test suites are left running
> when they should be deleted. By adding expiration dates, developers now
> have a chance to review the test suites based on latest status. The
> expiration date can be set to 'never' if necessary. An explanation is
> required for such a case.
>
> I will resent this a couple of times in case some people should miss this
> due to the holiday season.
>
> And happy holidays!
>
> thanks, Weizhong
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSir%2Biz%2BgNqkLsVQQDEhKQH2TnuX-u4OVNjYSZz6yNpEu-w%40mail.gmail.com.

[blink-dev] Re: PSA: Virtual web test suits now have expiration dates

['Weizhong Xia' via blink-dev]

For those who don't know already, the virtual test suites config file is
located at //third_party/blink/web_tests/VirtualTestSuites.

On Tue, May 30, 2023 at 10:23 AM Weizhong Xia  wrote:

> Hi Folks
>
> This is a reminder as we are approaching July 1st, 2023, the initial
> expiration date I set for all the virtual test suites. I did a search just
> now. Most VTS would still expire at that date (161 out of 194).
>
> Can we review this to make sure we do not lose coverage suddenly?
>
> thanks, Weizhong
>
> On Tue, Nov 22, 2022 at 12:07 PM Weizhong Xia  wrote:
>
>> Hi Folks
>>
>> TL;DR: We now support expiration dates for virtual test suites. All
>> virtual test suites now have an initial expiration date of July 1st, 2023.
>> If you own any virtual test suites, please review this value, and update
>> that to an appropriate date.
>>
>>  In the past we have seen cases where virtual test suites are left
>> running when they should be deleted. By adding expiration dates, developers
>> now have a chance to review the test suites based on latest status. The
>> expiration date can be set to 'never' if necessary. An explanation is
>> required for such a case.
>>
>> I will resent this a couple of times in case some people should miss this
>> due to the holiday season.
>>
>> And happy holidays!
>>
>> thanks, Weizhong
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSio1cBR%3D1DENPbzJVPgh%2BVgn3JG-gTx_p%3DC2hw6aq8MdNg%40mail.gmail.com.

[blink-dev] Fwd: An update to the Virtual Test Suite expiration policy

['Weizhong Xia' via blink-dev]

FYI

-- Forwarded message -
From: Weizhong Xia 
Date: Fri, Jun 23, 2023 at 1:38 PM
Subject: An update to the Virtual Test Suite expiration policy
To: Elly Fong-Jones , Mason Freed 
Cc: Andreas Haas , Reilly Grant ,
chromium-dev-internal 


Thanks to everyone who shared feedback/suggestions.

We have made a change on how expiration date is handled in this CL
. In
summary we will NOT automatically stop running virtual test suites on
expiration. Instead a P1 crbug will be filed against the owner when VTS
expires. A CL to remove the VTS will be created for review 30 days later if
the expiration date is not extended.

We are working on finding the owners for the VTSs. Before we find owners
for a VTS, it will continue to run as usual.

Thanks, Weizhong

On Wed, May 31, 2023 at 1:52 PM Elly Fong-Jones 
wrote:

> The approach the flag metadata file uses:
> https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/flag-metadata.json;l=12?q=flag-metadata.jso&ss=chromium
>
> has been serving us well so far :)
>
> -- elly
>
> On Wed, May 31, 2023 at 1:48 PM Andreas Haas  wrote:
>
>> I would prefer an owner's email address, ideally multiple email addresses
>> should be possible so that also the team's mailing list can be added.
>>
>> Cheers, Andreas
>>
>> On Wed, May 31, 2023 at 10:44 PM Weizhong Xia 
>> wrote:
>>
>>> Yes!
>>>
>>> Between an owner's email address and a crbug component, which one do you
>>> prefer? I am leaning toward letting the config file accept an optional
>>> component field, and we will automatically file a crbug when a VTS is about
>>> to expire. When no component is given, the crbug will be filed to an
>>> umbrella component, e.g. Test.
>>>
>>> thanks, Weizhong
>>>
>>> On Wed, May 31, 2023 at 10:54 AM Reilly Grant 
>>> wrote:
>>>
 For flag and histogram expiration we have owners assigned for each and
 will automatically email them when expiration is getting close. I notice
 that there are no owners listed for virtual test suites. Can we add this?
 [image: Google Logo]
 Reilly Grant
 Software Engineer
 reil...@google.com
 Google Chrome


 On Wed, May 31, 2023 at 9:47 AM Weizhong Xia 
 wrote:

> For those who don't know already, the virtual test suites config file
> is located at //third_party/blink/web_tests/VirtualTestSuites.
>
> On Tue, May 30, 2023 at 10:23 AM Weizhong Xia 
> wrote:
>
>> Hi Folks
>>
>> This is a reminder as we are approaching July 1st, 2023, the initial
>> expiration date I set for all the virtual test suites. I did a search 
>> just
>> now. Most VTS would still expire at that date (161 out of 194).
>>
>> Can we review this to make sure we do not lose coverage suddenly?
>>
>> thanks, Weizhong
>>
>> On Tue, Nov 22, 2022 at 12:07 PM Weizhong Xia 
>> wrote:
>>
>>> Hi Folks
>>>
>>> TL;DR: We now support expiration dates for virtual test suites. All
>>> virtual test suites now have an initial expiration date of July 1st, 
>>> 2023.
>>> If you own any virtual test suites, please review this value, and update
>>> that to an appropriate date.
>>>
>>>  In the past we have seen cases where virtual test suites are left
>>> running when they should be deleted. By adding expiration dates, 
>>> developers
>>> now have a chance to review the test suites based on latest status. The
>>> expiration date can be set to 'never' if necessary. An explanation is
>>> required for such a case.
>>>
>>> I will resent this a couple of times in case some people should miss
>>> this due to the holiday season.
>>>
>>> And happy holidays!
>>>
>>> thanks, Weizhong
>>>
>>> --
> You received this message because you are subscribed to the Google
> Groups "chromium-dev-internal" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to chromium-dev-internal+unsubscr...@google.com.
> To view this discussion on the web visit
> https://groups.google.com/a/google.com/d/msgid/chromium-dev-internal/CADXrSiokWve4vRXrKEkLNVHA3EvbDBT4QO1Rx3xaiY4Pib9mEA%40mail.gmail.com
> 
> .
>
 --
>>> You received this message because you are subscribed to the Google
>>> Groups "chromium-dev-internal" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to chromium-dev-internal+unsubscr...@google.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/a/google.com/d/msgid/chromium-dev-internal/CADXrSipQKTXD_g_FxaRcjcOjGgNXd5DSJfp-OiGtm_eVv427tg%40mail.gmail.com
>>> 

[blink-dev] PSA: migration to Wptrunner has started

['Weizhong Xia' via blink-dev]

TLDR: chrome_wpt_tests and headless_shell_wpt_tests have been added to
Chromium CI/CQ, and migration to Wptrunner has started.

Hello there

For quite some time web platform developers have wished to have a uniform
way to run web platform tests, which is to converge to Wptrunner. Blink
Engprod team has been working on this for quite some time, and now we are
happy to announce that the infra is ready and we can now start to migrate
the WPTs to run with Wptrunner, or run_wpt_tests.py, which is a wrapper
built on top of Wptrunner and aims to make it easier to use Wptrunner in
chromium.

Migration Plan

To migrate a directory to Wptrunner, you could simply add the directory to
//third_party/blink/web_tests/TestLists/headless_shell.filter/
,
and the tests in that directory will start to run with Wptrunner + Headless
shell on Chromium CI/CQ. Our plan is to fully migrate to Wptrunner by the *end
of the year*. By that time we will have three test suites for web tests, as
listed in the table below:

Suite Name

Browser Under Test

Harness

Tests Run

blink_web_tests

Content Shell

run_web_tests.py

Legacy web tests and wpt_internal

chrome_wpt_tests

Chrome new headless

run_wpt_tests.py

WPTs under external/wpt that depends on //chrome layer

headless_shell_wpt_tests

Headless Shell

run_wpt_tests.py

All other WPTs under external/wpt

How could you help?

The migration will be done directory by directory, and there are 2000+
regressions on each platform. We will start to create crbugs with the label
“wptrunner-migration” for those regressions. Please watch out for the
crbugs and help fix them so that we can finish the migration on time.

If you need any support during this process, the document is here
.
Also please feel free to reach out to weizhong@ or jonathanjlee@ if you
want to have a chat.

Thank you!

Weizhong on behalf of Blink Engprod

-- 
Empower web platform community with a single test runner that is fast,
stable and easy to use.

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirCEhXLO%3Dz%3DnhwCpjkfcNZ0Q8Ox-gnOBytUURp8OGReNg%40mail.gmail.com.

Re: [blink-dev] PSA: migration to Wptrunner has started

['Weizhong Xia' via blink-dev]

Right, performance is the major reason chrome new headless will not be used
for the majority of WPTs. One mitigation plan is to run all WPTs on CI with
chrome new headless, where it is less sensitive to the total test time.

On Wed, Sep 4, 2024 at 4:47 AM PhistucK  wrote:

> Sounds like headless shell will be used for the majority of the tests, but
> that is not testing the full browser, but just a headless shell which goes
> through different code paths...
> I realise that previously it was probably run using content shell or
> something like that which is also not as representative, are there any
> plans to actually use the full browser for running tests?
> If it is too slow, well... File bugs for making it faster ;)
>
> ☆*PhistucK*
>
>
> On Tue, Sep 3, 2024 at 7:10 PM 'Weizhong Xia' via blink-dev <
> blink-dev@chromium.org> wrote:
>
>> TLDR: chrome_wpt_tests and headless_shell_wpt_tests have been added to
>> Chromium CI/CQ, and migration to Wptrunner has started.
>>
>> Hello there
>>
>> For quite some time web platform developers have wished to have a uniform
>> way to run web platform tests, which is to converge to Wptrunner. Blink
>> Engprod team has been working on this for quite some time, and now we are
>> happy to announce that the infra is ready and we can now start to migrate
>> the WPTs to run with Wptrunner, or run_wpt_tests.py, which is a wrapper
>> built on top of Wptrunner and aims to make it easier to use Wptrunner in
>> chromium.
>>
>> Migration Plan
>>
>> To migrate a directory to Wptrunner, you could simply add the directory
>> to //third_party/blink/web_tests/TestLists/headless_shell.filter/
>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/TestLists/headless_shell.filter>,
>> and the tests in that directory will start to run with Wptrunner + Headless
>> shell on Chromium CI/CQ. Our plan is to fully migrate to Wptrunner by the 
>> *end
>> of the year*. By that time we will have three test suites for web tests,
>> as listed in the table below:
>>
>> Suite Name
>>
>> Browser Under Test
>>
>> Harness
>>
>> Tests Run
>>
>> blink_web_tests
>>
>> Content Shell
>>
>> run_web_tests.py
>>
>> Legacy web tests and wpt_internal
>>
>> chrome_wpt_tests
>>
>> Chrome new headless
>>
>> run_wpt_tests.py
>>
>> WPTs under external/wpt that depends on //chrome layer
>>
>> headless_shell_wpt_tests
>>
>> Headless Shell
>>
>> run_wpt_tests.py
>>
>> All other WPTs under external/wpt
>>
>> How could you help?
>>
>> The migration will be done directory by directory, and there are 2000+
>> regressions on each platform. We will start to create crbugs with the label
>> “wptrunner-migration” for those regressions. Please watch out for the
>> crbugs and help fix them so that we can finish the migration on time.
>>
>> If you need any support during this process, the document is here
>> <https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/run_web_platform_tests.md>.
>> Also please feel free to reach out to weizhong@ or jonathanjlee@ if you
>> want to have a chat.
>>
>> Thank you!
>>
>> Weizhong on behalf of Blink Engprod
>>
>> --
>> Empower web platform community with a single test runner that is fast,
>> stable and easy to use.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "blink-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to blink-dev+unsubscr...@chromium.org.
>> To view this discussion on the web visit
>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirCEhXLO%3Dz%3DnhwCpjkfcNZ0Q8Ox-gnOBytUURp8OGReNg%40mail.gmail.com
>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirCEhXLO%3Dz%3DnhwCpjkfcNZ0Q8Ox-gnOBytUURp8OGReNg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSiqZLOYC3AUKUPRkdMp8jGERx2dvDW6CeAqxVzdfy5biCQ%40mail.gmail.com.