Title: [213918] trunk/Tools
- Revision
- 213918
- Author
- commit-qu...@webkit.org
- Date
- 2017-03-14 11:22:06 -0700 (Tue, 14 Mar 2017)
Log Message
Nwtr ignores ImageDiff's errors for ref tests
https://bugs.webkit.org/show_bug.cgi?id=168033
Patch by Fujii Hironori <hironori.fu...@sony.com> on 2017-03-14
Reviewed by Alexey Proskuryakov.
Nwtr checks ImageDiff's errors only for pixel tests, but for ref
tests. Those errors of ref tests also should be checked.
In the current implementation of expected mismatch ref tests,
diff_image was called if the image hashes match. This is useless
because two images are ensured identical in that case. Calling
image_hash is considered unnecessary for expected mismatch ref
tests. Do not call diff_image for them.
As the result, check the error only for expected match ref tests.
* Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
(SingleTestRunner._compare_image): Rename a variable 'err_str' to 'error_string'.
(SingleTestRunner._compare_output_with_reference): Do not call
diff_image for expected mismatch ref tests. Check the error and
marked the test failed for expected match ref tests.
Modified Paths
Diff
Modified: trunk/Tools/ChangeLog (213917 => 213918)
--- trunk/Tools/ChangeLog 2017-03-14 18:19:47 UTC (rev 213917)
+++ trunk/Tools/ChangeLog 2017-03-14 18:22:06 UTC (rev 213918)
@@ -1,3 +1,27 @@
+2017-03-14 Fujii Hironori <hironori.fu...@sony.com>
+
+ Nwtr ignores ImageDiff's errors for ref tests
+ https://bugs.webkit.org/show_bug.cgi?id=168033
+
+ Reviewed by Alexey Proskuryakov.
+
+ Nwtr checks ImageDiff's errors only for pixel tests, but for ref
+ tests. Those errors of ref tests also should be checked.
+
+ In the current implementation of expected mismatch ref tests,
+ diff_image was called if the image hashes match. This is useless
+ because two images are ensured identical in that case. Calling
+ image_hash is considered unnecessary for expected mismatch ref
+ tests. Do not call diff_image for them.
+
+ As the result, check the error only for expected match ref tests.
+
+ * Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
+ (SingleTestRunner._compare_image): Rename a variable 'err_str' to 'error_string'.
+ (SingleTestRunner._compare_output_with_reference): Do not call
+ diff_image for expected mismatch ref tests. Check the error and
+ marked the test failed for expected match ref tests.
+
2017-03-14 Brady Eidson <beid...@apple.com>
REGRESSION (r213877): WebKit2.CookieManager fails.
Modified: trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py (213917 => 213918)
--- trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py 2017-03-14 18:19:47 UTC (rev 213917)
+++ trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py 2017-03-14 18:22:06 UTC (rev 213918)
@@ -273,11 +273,11 @@
failures.append(test_failures.FailureMissingImageHash())
elif driver_output.image_hash != expected_driver_output.image_hash:
diff_result = self._port.diff_image(expected_driver_output.image, driver_output.image)
- err_str = diff_result[2]
- if err_str:
- _log.warning(' %s : %s' % (self._test_name, err_str))
+ error_string = diff_result[2]
+ if error_string:
+ _log.warning(' %s : %s' % (self._test_name, error_string))
failures.append(test_failures.FailureImageHashMismatch())
- driver_output.error = (driver_output.error or '') + err_str
+ driver_output.error = (driver_output.error or '') + error_string
else:
driver_output.image_diff = diff_result[0]
if driver_output.image_diff:
@@ -331,17 +331,19 @@
if not reference_driver_output.image_hash and not actual_driver_output.image_hash:
failures.append(test_failures.FailureReftestNoImagesGenerated(reference_filename))
elif mismatch:
+ # Calling image_hash is considered unnecessary for expected mismatch ref tests.
if reference_driver_output.image_hash == actual_driver_output.image_hash:
- diff_result = self._port.diff_image(reference_driver_output.image, actual_driver_output.image, tolerance=0)
- if not diff_result[0]:
- failures.append(test_failures.FailureReftestMismatchDidNotOccur(reference_filename))
- else:
- _log.warning(" %s -> ref test hashes matched but diff failed" % self._test_name)
-
+ failures.append(test_failures.FailureReftestMismatchDidNotOccur(reference_filename))
elif reference_driver_output.image_hash != actual_driver_output.image_hash:
+ # ImageDiff has a hard coded color distance threshold even though tolerance=0 is specified.
diff_result = self._port.diff_image(reference_driver_output.image, actual_driver_output.image, tolerance=0)
- if diff_result[0]:
+ error_string = diff_result[2]
+ if error_string:
+ _log.warning(' %s : %s' % (self._test_name, error_string))
failures.append(test_failures.FailureReftestMismatch(reference_filename))
+ actual_driver_output.error = (actual_driver_output.error or '') + error_string
+ elif diff_result[0]:
+ failures.append(test_failures.FailureReftestMismatch(reference_filename))
else:
_log.warning(" %s -> ref test hashes didn't match but diff passed" % self._test_name)
_______________________________________________
webkit-changes mailing list
webkit-changes@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-changes
