Hi Blink devs,

The move has landed (https://crrev.com/1039971).

If you have any pending CLs that add/remove/modify files under
web_tests/platform/generic, please sync, and add/remove/modify the
corresponding file under web_tests/ instead.

Thanks,
Xianzhu

On Fri, Aug 26, 2022 at 10:57 AM Weizhong Xia <[email protected]> wrote:

> Hi Blink devs
>
> FYI we are finally ready to move generic baselines back to their original
> places: the test folders. I plan to land the CL today. To avoid any merge
> conflict during this process, we will add an OWNERS file to
> //third_party/blink/web_tests/platform/generic. Any change to that folder
> will require an additional +1, and will not be approved. Once the move is
> done, the "platform/generic" folder will be removed.
>
> thanks, Weizhong
>
> On Thu, Jun 23, 2022 at 11:00 AM Weizhong Xia <[email protected]> wrote:
>
>> Hi blink devs
>>
>> Thanks to those who joined the survey at
>> https://forms.gle/ju45qciS5VTR4ywN7. Most of you expressed the desire to
>> put baselines at the same place of the tests. Your voice is heard, and here
>> is the plan for the next step.
>>
>> In Q3 we will work on to completely separate legacy layout tests and wpt
>> tests, to put them under `third_party/blink/web_tests` and
>> `third_party/blink/wpt_tests` respectively. Generic baselines (including
>> generic virtual baselines) will be moved back to their previous place.
>> Rebaseline tool will be updated to work with this structure, and update
>> baselines for legacy layout tests and wpt tests in a single run. We will
>> have different copies of *TestExpectations*, *FlagSpecificConfig*,
>> *VirtualTestSuites* etc for legacy tests and wpt tests. When working on
>> those files, we will need to make sure we are updating the correct copy of
>> the file. (We will investigate if we need some presubmit check for such a
>> scenario).
>>
>> The reason for this is two fold: as requested we want to put baselines
>> side to side to the tests, and we want to make the directory structure
>> right to speed up the switch to wptrunner.
>>
>> Thoughts? Feel free to leave a comment in crbug/1299834
>> <https://crbug.com/1299834>.
>>
>> Thanks, Weizhong
>>
>>
>>
>> On Mon, Jun 6, 2022 at 5:00 PM Xianzhu Wang <[email protected]>
>> wrote:
>>
>>> On Mon, Jun 6, 2022 at 3:52 PM Weizhong Xia <[email protected]> wrote:
>>>
>>>> Xianzhu, yes 'baselines' is the name we agreed on previously. The
>>>> reason I later changed back to use 'platform' is because that will make the
>>>> CL smaller, and make it easier for gerrit to handle it. We can make one
>>>> round rename when everything is stabilized. (I left you a message when you
>>>> are OOO. I am not sure if that message lived long enough for you to catch
>>>> it.)
>>>>
>>>
>>> I caught the message. I guessed that the name 'platform' might be one of
>>> the reasons for the surprise to blink developers after the change, and the
>>> name 'baselines' might make the change easier to explain :)
>>>
>>> On Mon, Jun 6, 2022 at 3:52 PM Domenic Denicola <[email protected]>
>>> wrote:
>>>
>>>> I'm not sure I understand this logic. -expected.txt files are just
>>>> more-convenient, more in-your-face versions of TestExpectations files.
>>>> Surely you're not suggesting we get rid of TestExpectations files?
>>>>
>>>
>>> Sorry, "-expected.txt ... should be ... eventually be removed" in my
>>> previous email was not clear. I meant for each individual -expected.txt we
>>> should eventually remove it because we should fix the failure. The same
>>> logic applies to TestExpectations. At any time we may allow a certain
>>> number of failures but we should keep the number as small as possible.
>>>
>>> I think we should prefer TestExpectations to -expected.txt for WPT tests
>>> because the entries in TestExpectations have associated bugs which track
>>> the fixing process, unless we find a better way to track the fixing of the
>>> failures in -expected.txt. -expected.txt files do have their values, e.g.
>>> for partially-passing tests we can discover regressions and progressions of
>>> individual sub tests, but they should be rare.
>>>
>>> I think separating -expected.txt from the tests has the following
>>> benefits:
>>> - It makes it clear to blink developers that the files are not a part of
>>> WPT.
>>> - It simplifies the WPT export/import process and others by reducing
>>> blink-specific files under external/wpt.
>>>
>>> It does make it more difficult to find -expected.txt, but we already
>>> have the similar well-known logic for platform-specific baselines. Though
>>> platform-specific baselines are rare, ignoring a platform baseline can
>>> still cause surprises.
>>>
>>> I think we can improve the test result viewer
>>> <https://test-results.appspot.com/data/layout_results/Mac10_15_Tests/26148/blink_web_tests/layout-test-results/results.html>
>>> - to better show -expected.txt for passing tests
>>> - to show information about tests without actually running the tests
>>> WDYT?
>>>
>>>
>>>>
>>>> On Mon, Jun 6, 2022 at 6:30 PM Xianzhu Wang <[email protected]>
>>>> wrote:
>>>>
>>>>> I think we first need to answer a question: Why do we need
>>>>> *-expected.txt for WPT tests?
>>>>>
>>>>> Upstream WPT doesn't have *-expected.txt. *-expected.txt is a
>>>>> blink-specific thing to allow some failing WPT tests to pass on blink with
>>>>> temporarily allowed failures. If a test has -expected.txt, it means Chrome
>>>>> behaves differently than the standard web platform behavior (or the test
>>>>> itself is wrong). The files are sometimes harmful because they hide
>>>>> failures and web platform incompatibilities (e.g.
>>>>> <http://crbug.com/772405>). So I think -expected.txt files for WPT
>>>>> tests should be rare and eventually be removed. An -expected.txt should 
>>>>> not
>>>>> be treated as a part of the test itself because a) it doesn't exist in
>>>>> upstream WPT and 2) it doesn't describe the standard expected behavior,
>>>>> thus isn't necessarily placed besides the test.
>>>>>
>>>>> The directory name 'platform' may be misleading. 'baselines' may be a
>>>>> better name (but we should not rename until we decide what to do for
>>>>> generic baselines). For WPT tests, 'failures' is perhaps an even better
>>>>> name.
>>>>>
>>>>> On Friday, June 3, 2022 at 6:55:49 AM UTC-7 Dominic Farolino wrote:
>>>>>
>>>>>> For the long term, we can definitely move this back once we have
>>>>>>> separate legacy tests and wpt into different folders. I can also reach 
>>>>>>> out
>>>>>>> to you guys to better understand your needs.
>>>>>>>
>>>>>>> For the folders Dom mentioned, I can check to see if any of that can
>>>>>>> be removed. I understand the README is needed for virtual test suites.
>>>>>>>
>>>>>>
>>>>>> Can you define "long term" here? Is there a timeline? I was really
>>>>>> hoping that we'd go back to how things were almost immediately. Still the
>>>>>> need for the delay does not quite make sense to me, and I'm really
>>>>>> hoping that for Blink developer experience we can revert this back to the
>>>>>> previous setup ASAP.
>>>>>>
>>>>>> Under "web_tests/virtual" we have baselines for wpt and pure virtual
>>>>>>> legacy tests, we can list the folders under "virtual/prefix" for each
>>>>>>> virtual test suite, which will make the dependency very large(yes), and 
>>>>>>> we
>>>>>>> need make such change each time when we add new virtual suites. I don't
>>>>>>> think this is something blink devs want to do.
>>>>>>>
>>>>>>
>>>>>> I'm sorry, but like Domenic I am not sure what to make of much of
>>>>>> this. What I do know is that Blink developers also don't want the current
>>>>>> WPT writing experience where expectation files are positioned far away 
>>>>>> from
>>>>>> the source test. I think this experience matters a lot.
>>>>>>
>>>>>> On Thu, Jun 2, 2022 at 4:25 PM Weizhong Xia <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Previously you don't need to specify anything in BUILD.gn is because
>>>>>>> we are downloading the whole "web_tests" folder. Now we want to run 
>>>>>>> legacy
>>>>>>> tests and wpt in different steps. Under "web_tests/virtual" we have
>>>>>>> baselines for wpt and pure virtual legacy tests, we can list the folders
>>>>>>> under "virtual/prefix" for each virtual test suite, which will make the
>>>>>>> dependency very large(yes), and we need make such change each time when 
>>>>>>> we
>>>>>>> add new virtual suites. I don't think this is something blink devs want 
>>>>>>> to
>>>>>>> do.
>>>>>>>
>>>>>>> The discussion is at the beginning of the crbug. So pls scroll back
>>>>>>> to #c1, and read from there.
>>>>>>>
>>>>>>> thanks, Weizhong
>>>>>>>
>>>>>>> On Thu, Jun 2, 2022 at 12:29 PM Domenic Denicola <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 2, 2022 at 3:21 PM Weizhong Xia <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Folks
>>>>>>>>>
>>>>>>>>> I'm sorry to see this has caused inconvenience, and sorry for
>>>>>>>>> being late in response to this, due to the same reason Dom had.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for responding and listening to our concerns!
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> The reason to move baselines to one central place at this point is
>>>>>>>>> to make it possible to specify dependency in BUILD.gn.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I don't quite understand this. I've had to work with WPTs and WPT
>>>>>>>> expectations my entire time working on Chromium. I've never had to 
>>>>>>>> "specify
>>>>>>>> dependency in BUILD.gn"; I don't really know what that means. I'd love 
>>>>>>>> to
>>>>>>>> hear more (or be referred to a doc explaining the issue), so that I
>>>>>>>> understand why we're making the sacrifice we're making. (The linked bug
>>>>>>>> isn't very understandable for me, unfortunately. I can't even 
>>>>>>>> understand
>>>>>>>> enough to find the part you mentioned where you discussed different
>>>>>>>> approaches.)
>>>>>>>>
>>>>>>>> I'm also unsure how the decision was weighed. Is the population of
>>>>>>>> people specifying dependency in BUILD.gn very large, so that their 
>>>>>>>> needs
>>>>>>>> are outweighing those of the Chromium developers working with web 
>>>>>>>> platform
>>>>>>>> tests?
>>>>>>>>
>>>>>>>>
>>>>>>>>> This is part of the work to use upstream wptrunner to run wpt
>>>>>>>>> tests. In crbug/1299834 <https://crbug.com/1299834> we have tried
>>>>>>>>> to discuss some different approaches. I would say this is the least
>>>>>>>>> disruptive way. For blink engprod, I think to make blink devs happy 
>>>>>>>>> is our
>>>>>>>>> top priority.
>>>>>>>>>
>>>>>>>>> For the long term, we can definitely move this back once we have
>>>>>>>>> separate legacy tests and wpt into different folders. I can also 
>>>>>>>>> reach out
>>>>>>>>> to you guys to better understand your needs.
>>>>>>>>>
>>>>>>>>> For the folders Dom mentioned, I can check to see if any of that
>>>>>>>>> can be removed. I understand the README is needed for virtual test 
>>>>>>>>> suites.
>>>>>>>>>
>>>>>>>>> Cheers, Weizhong
>>>>>>>>>
>>>>>>>>> On Wednesday, June 1, 2022 at 6:06:56 AM UTC-7
>>>>>>>>> [email protected] wrote:
>>>>>>>>>
>>>>>>>>>> +1 from me as well. I was similarly caught by surprise by this
>>>>>>>>>> change (during reviews for webexposed changes), and am similarly not 
>>>>>>>>>> seeing
>>>>>>>>>> the upside for this.
>>>>>>>>>>
>>>>>>>>>> While I'm sure this is a change that was meant to be a positive
>>>>>>>>>> one, I'd love to better understand the reasoning, and whether the 
>>>>>>>>>> current
>>>>>>>>>> situation is a temporary one, or one that is planned to be permanent 
>>>>>>>>>> even
>>>>>>>>>> after the move to blink_wpt_tests is done.
>>>>>>>>>>
>>>>>>>>>> On Fri, May 27, 2022 at 6:10 PM Domenic Denicola <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1. This was really unpleasantly surprising. When I first saw
>>>>>>>>>>> the original blink-dev email, I thought "generic baselines" meant 
>>>>>>>>>>> something
>>>>>>>>>>> like "non-web platform test baselines", not "WPT expectation files 
>>>>>>>>>>> that are
>>>>>>>>>>> platform-agnostic".
>>>>>>>>>>>
>>>>>>>>>>> In addition to the context-switching cost, it's just much harder
>>>>>>>>>>> to navigate between tests and their expectations, which is 
>>>>>>>>>>> something I do
>>>>>>>>>>> quite often. E.g., they are no longer grouped together in code 
>>>>>>>>>>> reviews,
>>>>>>>>>>> since their file paths are lexicographically far away from each 
>>>>>>>>>>> other. And,
>>>>>>>>>>> as someone maintaining and reviewing several WPT directories, 
>>>>>>>>>>> moving these
>>>>>>>>>>> crucial files out of the directories I commonly work in (and have 
>>>>>>>>>>> metadata
>>>>>>>>>>> marking me as the point-of-contact for) into separate directories 
>>>>>>>>>>> dilutes
>>>>>>>>>>> the cohesiveness of my projects.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 27, 2022 at 12:03 PM Dominic Farolino <
>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I write a lot of web platform tests as a Web Platform engineer;
>>>>>>>>>>>> recently I wrote one in external/wpt/
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/>
>>>>>>>>>>>> (the external web platform tests directory), and was shocked to 
>>>>>>>>>>>> find the
>>>>>>>>>>>> eradication of `-expected.txt` files. I placed my expectations 
>>>>>>>>>>>> file next to
>>>>>>>>>>>> the source file as we've done for many years, and found that my 
>>>>>>>>>>>> test was
>>>>>>>>>>>> "failing" because the test runner couldn't find my test 
>>>>>>>>>>>> expectations file.
>>>>>>>>>>>>
>>>>>>>>>>>> I dug deeper and found https://crrev.com/c/3603221 which was
>>>>>>>>>>>> responsible for moving more than 21,000 *platform-agnostic*
>>>>>>>>>>>> test expectations files away from their source files and into
>>>>>>>>>>>> web_tests/platform/generic
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/platform/generic/>
>>>>>>>>>>>> directory. I found more discussion in this email thread
>>>>>>>>>>>> <https://groups.google.com/a/chromium.org/g/blink-dev/c/0WmmgEkqdOo>
>>>>>>>>>>>>  which
>>>>>>>>>>>> I missed because blink-dev emails do not go directly in my inbox.
>>>>>>>>>>>>
>>>>>>>>>>>> I must say I find this change extraordinarily inconvenient as a
>>>>>>>>>>>> Web Platform engineer, and I want to push back against this. A 
>>>>>>>>>>>> minority of
>>>>>>>>>>>> web platform tests have platform-specific failures, which 
>>>>>>>>>>>> justifies the
>>>>>>>>>>>> need for *some* platform-specific test expectations
>>>>>>>>>>>> directories, but I believe a huge majority have generic baselines 
>>>>>>>>>>>> that are
>>>>>>>>>>>> wildly convenient to have right next to the actual tests 
>>>>>>>>>>>> themselves.
>>>>>>>>>>>> Putting them in a separate directory means I and others have to 
>>>>>>>>>>>> open a
>>>>>>>>>>>> separate browser tab to view how many expectations there are for a 
>>>>>>>>>>>> given
>>>>>>>>>>>> directory, and requires a lot of unnecessary context switching. It 
>>>>>>>>>>>> is
>>>>>>>>>>>> particularly confusing for clusters of tests whose names are all
>>>>>>>>>>>> *very* similar and vary by only a few numbers or suffixes—this
>>>>>>>>>>>> increases the cost of the context switching.
>>>>>>>>>>>>
>>>>>>>>>>>> Furthermore, it renders tons of directories absolutely useless!
>>>>>>>>>>>> All ~150 directories in web_tests/virtual
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/virtual/>
>>>>>>>>>>>>  (for
>>>>>>>>>>>> VirtualTestSuites) are just empty directories with README 
>>>>>>>>>>>> files—these used
>>>>>>>>>>>> to house virtualtest-specific expectations. So now for fenced 
>>>>>>>>>>>> frames (the
>>>>>>>>>>>> project I'm working on right now), we have the following test 
>>>>>>>>>>>> directories:
>>>>>>>>>>>>
>>>>>>>>>>>>    - web_tests/wpt_internal/fenced_frame/
>>>>>>>>>>>>    
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/wpt_internal/fenced_frame/>
>>>>>>>>>>>>    - web_tests/virtual/fenced-frame-mparch/
>>>>>>>>>>>>    
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/virtual/fenced-frame-mparch/>
>>>>>>>>>>>>    - web_tests/virtual/fenced-frame-shadow-dom/
>>>>>>>>>>>>    
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/virtual/fenced-frame-shadow-dom/>
>>>>>>>>>>>>    - web_tests/platform/generic/wpt_internal/fenced_frame/
>>>>>>>>>>>>    
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/platform/generic/wpt_internal/fenced_frame/>
>>>>>>>>>>>>    - web_tests/platform/generic/virtual/fenced-frame-mparch/
>>>>>>>>>>>>    
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/platform/generic/virtual/fenced-frame-mparch/>
>>>>>>>>>>>>    -
>>>>>>>>>>>>    web_tests/platform/generic/virtual/fenced-frame-shadow-dom/
>>>>>>>>>>>>    
>>>>>>>>>>>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/platform/generic/virtual/fenced-frame-shadow-dom/>
>>>>>>>>>>>>    - + platform-specific directories, which are relatively
>>>>>>>>>>>>    rare for us
>>>>>>>>>>>>
>>>>>>>>>>>> This is so weird! Regardless of whether or not there are plans
>>>>>>>>>>>> to clean this up, I can't see the upsides. The ousting of 
>>>>>>>>>>>> platform-agnostic
>>>>>>>>>>>> expectations is only an inconvenience for WP engineers, while there
>>>>>>>>>>>> might be some test-infra conveniences around BUILD.gn
>>>>>>>>>>>> dependencies
>>>>>>>>>>>> <https://bugs.chromium.org/p/chromium/issues/detail?id=1299834#:~:text=This%20way%20all%20the%20baselines%20will%20be%20in%20platform%20directory%2C%20make%20it%20a%20little%20bit%20easier%20to%20specify%20dependency%20in%20BUILD.gn.>
>>>>>>>>>>>>  (maybe?).
>>>>>>>>>>>> In any case, I am hard pressed to find justification in this move, 
>>>>>>>>>>>> and
>>>>>>>>>>>> would love to see if we can reconsider this.
>>>>>>>>>>>>
>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>
>>>>>>>>>>>> Dom
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>> Google Groups "blink-dev" group.
>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>> it, send an email to [email protected].
>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP-uykAN06y5o-WYznnicvm1YREbSsLbs6dM57LtL4vCWB%3Duzw%40mail.gmail.com
>>>>>>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP-uykAN06y5o-WYznnicvm1YREbSsLbs6dM57LtL4vCWB%3Duzw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "blink-dev" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to [email protected].
>>>>>>>>>>>
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAM0wra9jXeotZVYNKBMmW90x36%2BdOCqcqfZ-ZpPW0qJVUBptbQ%40mail.gmail.com
>>>>>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAM0wra9jXeotZVYNKBMmW90x36%2BdOCqcqfZ-ZpPW0qJVUBptbQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADBxridcOZZ_Dnpi5dxRV_QMVL3qFFzTbth43RAaJxj6qRt7KQ%40mail.gmail.com.

Reply via email to