Nice results, I hope we will be able to unksip a lot of tests. The guys
here are working on it right now ;-)
One problem with this could be that even if some tests are passing in a
particular directory, unskipping them can sometime cause other tests in
other directories to fail. It should not be too frequent but it means
you need to rerun the tests to check it. And of course the boring issue
of divergent results between the bot and other systems. So far we didn't
take much effort in reducing the skip list because even mantaining bot
greenness (with continuosly growing skip lists) needs a lot of efforts,
but it's time to change on it!
-KBalazs
On 11/08/2011 11:17 PM, Jesus Sanchez-Palencia wrote:
Hi folks,
Today I spent some time taking a look at the status of our Layout
Tests. After taking a look at the Skipped lists I decided to give it a
try to two largely skipped folders: http/tests/ and media/ .
I used: "run-webkit-tests --no-launch-safari --no-sample-on-timeout
--no-new-test-results --results-directory layout-test-results
--use-remote-links-to-tests --release --qt --force FOLDER", which was
copied from the Szeged bots.
I ran the tests for WebKit1 (6582a115083311f52297cde636c82c96c97f9f29)
with Qt 4.8 (6582a115083311f52297cde636c82c96c97f9f29) and for WebKit2
with Qt5 (6df6e99be09731991a1689cf1a71c601f4814865). For WK2 I ran it
on 2 different machines, running different Linux distros.
Perhaps I'm missing something or I did something wrong but, without
_any_ fix to WebKit nor Qt, and for my surprise, I manage to get a
reasonable amount of "Expected to fail, but passed" tests:
- media/ WK2: ~110 of "unexpected-passing" tests.
- http/tests/ WK2: ~120 of "unexpected-passing" tests.
- media/ WK1: 68 "unexpected-passing" tests.
- http/tests/ WK1: 45 "unexpected-passing" tests.
Which, according to Ossy, are Skipped tests that were marked as fail,
but that can be passing now.
Here you can find the list of unexpected-passing tests for WK2:
http://pastebin.com/zBGHcYa3
Here you can find the list of unexpected-passing tests for WK1:
http://pastebin.com/ZyMpK2T1
Since I don't have access to a proper buildbot environment, I wonder
if it wouldn't be possible for the Szeged folks to take a look at this
list and try to unskip these tests. Perhaps just running the same
command as I did might spot a certain amount of tests that could be
unskipped right now. I would be surprise if there isn't at least 10
tests among these ~200 that are passing already, not to say more...
Moreover, I believe this is a good way to easily spot tests that could
be passing; Just running the tests with --force to a certain folder
and then cross-checking the output list marked as "Expected to fail,
but passed". I was told this is the "Skipped list reducer week at
Szeged", so I hope you guys can use this somehow.
Cheers,
jesus
_______________________________________________
webkit-qt mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt
_______________________________________________
webkit-qt mailing list
[email protected]
http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt
