I see the CL changed the expectation of some tests from [ Timeout ] to [ Slow ], and the tests still timeout. To make the bot green, we need to choose from: a. Add back the [ Timeout ] entries in TestExpectations with the [ Slow ] entries in SlowTests; b. Revert the change for the tests (i.e. keep the [ Timeout ] entries in TestExpectations without [ Slow ] entries in SlowTests); c. Add [ Skip ] for the test. We can keep both [ Timeout ] and [ Slow ] as they don't matter if there is [ Skip ], but they partly describe the reason for the skip.
I think Weizhong's proposal is to handle #a which makes the tests unnecessarily take a long time before timeout. Proposal #1 changes #a to #c, and proposal #2 disallows #a. For now it's not easy to check if a timing-out test is just slow or it hangs indefinitely. @Brian Sheedy <bshe...@google.com> is working on a project <http://crbug.com/1222827> to automatically update TestExpectations based on bot history by removing stale failure/flaky entries for passing tests, to recover test coverage. I think the project could be extended in the future to handle slow/timeout entries based on bot history. For example, we can force running tests with Skip, and try bigger timeout for Timeout, on some bots periodically, to check 1) if a timing-out test is just slow, and 2) if a skipped test can be unskipped, and update test expectations automatically. On Thu, Aug 19, 2021 at 7:42 AM Stephen Chenney <schen...@chromium.org> wrote: > > > On Wed, Aug 18, 2021 at 3:06 PM Weizhong Xia <weizh...@google.com> wrote: > >> >> I recently found a test that was TIMEOUT in TestExpectations, and >> still timed out in Slow, so I left it in TestExpectations under the >> assumption that it will timeout faster there. >> What do you mean "timed out in Slow"? A test did not mark as slow, but >> timeout after 60 seconds? Worth mentioning some slow wpt tests are given >> the same timeout value as slow tests in the code. >> > > See https://chromium-review.googlesource.com/c/chromium/src/+/3097709/1 > and the blink_win7 try job. > > I moved the expectation > for external/wpt/css/css-fonts/test_font_family_parsing.html from the > TestExpectations file to the SlowTests file and ran the try jobs. The test > was still slower than the slow test timeout, and given the console output I > strongly suspect the test hangs indefinitely. > > >> But I did see some tests timed out at some abnormal value. For example >> tests below timeouts at 22 second. Not sure how this could happen. >> 19:13:50.493 20120 [3522/4178] >> external/wpt/webrtc-identity/idlharness.https.window.html >> failed unexpectedly (test timed out) 22.3300s >> >> On Wed, Aug 18, 2021 at 11:26 AM Stephen Chenney <schen...@chromium.org> >> wrote: >> >>> >>> >>> On Wed, Aug 18, 2021 at 12:01 PM 'Weizhong Xia' via blink-dev < >>> blink-dev@chromium.org> wrote: >>> >>>> Dear blink-devs, >>>> >>>> As another effort trying to speed up blink_web_tests on CQ, We are >>>> looking at slow tests that also timeouts. From the timing stats(example >>>> <https://test-results.appspot.com/data/layout_results/linux-rel/773524/blink_web_tests%20%28with%20patch%29/layout-test-results/results.html>) >>>> and some research, there are about 0.25% such tests, but running such tests >>>> takes about 4% of the total run time. So we are thinking of >>>> doing something about it. >>>> >>>> We would like to propose as below, and get feedback on this: >>>> 1. We add Skip to test expectations for slow tests that also timeouts. >>>> This will effectively skip all such tests on swarming bots, and save ~50s >>>> real time on linux-rel. >>>> 2. We add a presubmit check to make any unskipped Slow+Timeout an >>>> error. (We can also make this a warning, but such warning will persist, so >>>> kind of annoying) >>>> 3. Devs are encouraged to take actions to either fix the timeout, or >>>> break the slow tests into pieces to make it not slow. >>>> >>>> >>> I recently found a test that was TIMEOUT in TestExpectations, and still >>> timed out in Slow, so I left it in TestExpectations under the assumption >>> that it will timeout faster there. >>> >>> The underlying issue is not a slow test but a test that hangs and we may >>> or may not want to skip such a test. Would identifying these cases and >>> moving them from Slow to TestExpectations have any noticeable impact? >>> >>> >>>> Please let us know your opinions on this. >>>> >>>> Thanks, Weizhong >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "blink-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to blink-dev+unsubscr...@chromium.org. >>>> To view this discussion on the web visit >>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipU3NDSp7tWd%3DqA%3DrZWMrmXjfT4NzTG7U8EoEAgsQ1QXw%40mail.gmail.com >>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipU3NDSp7tWd%3DqA%3DrZWMrmXjfT4NzTG7U8EoEAgsQ1QXw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADBxrifVew8_iO-GzLzgAavX3_ZET50v_rJJ7Fqw49JErdZQcw%40mail.gmail.com.
