On Wed, Aug 18, 2021 at 3:06 PM Weizhong Xia <weizh...@google.com> wrote:
> >> I recently found a test that was TIMEOUT in TestExpectations, and still > timed out in Slow, so I left it in TestExpectations under the assumption > that it will timeout faster there. > What do you mean "timed out in Slow"? A test did not mark as slow, but > timeout after 60 seconds? Worth mentioning some slow wpt tests are given > the same timeout value as slow tests in the code. > See https://chromium-review.googlesource.com/c/chromium/src/+/3097709/1 and the blink_win7 try job. I moved the expectation for external/wpt/css/css-fonts/test_font_family_parsing.html from the TestExpectations file to the SlowTests file and ran the try jobs. The test was still slower than the slow test timeout, and given the console output I strongly suspect the test hangs indefinitely. > But I did see some tests timed out at some abnormal value. For example > tests below timeouts at 22 second. Not sure how this could happen. > 19:13:50.493 20120 [3522/4178] > external/wpt/webrtc-identity/idlharness.https.window.html > failed unexpectedly (test timed out) 22.3300s > > On Wed, Aug 18, 2021 at 11:26 AM Stephen Chenney <schen...@chromium.org> > wrote: > >> >> >> On Wed, Aug 18, 2021 at 12:01 PM 'Weizhong Xia' via blink-dev < >> blink-dev@chromium.org> wrote: >> >>> Dear blink-devs, >>> >>> As another effort trying to speed up blink_web_tests on CQ, We are >>> looking at slow tests that also timeouts. From the timing stats(example >>> <https://test-results.appspot.com/data/layout_results/linux-rel/773524/blink_web_tests%20%28with%20patch%29/layout-test-results/results.html>) >>> and some research, there are about 0.25% such tests, but running such tests >>> takes about 4% of the total run time. So we are thinking of >>> doing something about it. >>> >>> We would like to propose as below, and get feedback on this: >>> 1. We add Skip to test expectations for slow tests that also timeouts. >>> This will effectively skip all such tests on swarming bots, and save ~50s >>> real time on linux-rel. >>> 2. We add a presubmit check to make any unskipped Slow+Timeout an error. >>> (We can also make this a warning, but such warning will persist, so kind of >>> annoying) >>> 3. Devs are encouraged to take actions to either fix the timeout, or >>> break the slow tests into pieces to make it not slow. >>> >>> >> I recently found a test that was TIMEOUT in TestExpectations, and still >> timed out in Slow, so I left it in TestExpectations under the assumption >> that it will timeout faster there. >> >> The underlying issue is not a slow test but a test that hangs and we may >> or may not want to skip such a test. Would identifying these cases and >> moving them from Slow to TestExpectations have any noticeable impact? >> >> >>> Please let us know your opinions on this. >>> >>> Thanks, Weizhong >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to blink-dev+unsubscr...@chromium.org. >>> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipU3NDSp7tWd%3DqA%3DrZWMrmXjfT4NzTG7U8EoEAgsQ1QXw%40mail.gmail.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSipU3NDSp7tWd%3DqA%3DrZWMrmXjfT4NzTG7U8EoEAgsQ1QXw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGsbWzTnrBDp5OPuAef%3DH1q8WBRRvo7kW_tgjaqFysGvV7JnmQ%40mail.gmail.com.
