On 02/02/16 00:27, Ryosuke Niwa wrote: > On Mon, Feb 1, 2016 at 7:32 AM, Carlos Alberto Lopez Perez > <clo...@igalia.com> wrote: >> On 31/01/16 05:16, Filip Pizlo wrote: >>> As my original message said, I was wondering if there was any support >>> for running some JavaScript tests *in browser*. run-jsc-stress-tests >>> doesn’t support that because it doesn’t know what a browser is. >>> >>> Some tests, like Kraken, Octane, JetStream, and Speedometer, either >>> require a browser to run (like JetStream and Speedometer) or have >>> significantly different behavior in the browser than in their >>> command-line harnesses (like Kraken and Octane). If you did have a >>> bot that ran these tests in some GTK+ or EFL browser, you’d probably >>> catch bugs that testing the JSC shell cannot catch. >>> >>> -Filip >> >> Some questions regarding this JSC in-browser benchmarks: >> >> How does the Apple port runs this? Do you use some script that is >> currently available on the WebKit source tree? Are the buildbots running >> this tests public? > > We've been running those benchmarks with > http://trac.webkit.org/browser/trunk/Tools/Scripts/run-benchmark >
I see. But this script seems focused on comparing the performance between different browsers (safari vs chrome vs firefox) rather than in testing and comparing the performance between different revisions of WebKit. Do you think it makes any difference (from the point of view of detecting failures, not from the performance PoV) to run this tests in a full-fledged browser like Safari rather than in WebKitTestRunner ? We already have a performance test bot running tests inside WTR. And I see that the current set of tests executed on this bot already includes Speedometer, and that JetStream and Sunspider are skipped on PerformanceTests/Skipped. So I see some options going forward: - Fix the JetStream and Sunspider tests so they can be run as part of the current run-perf-tests script that the performance bots execute. - Implement support on the script run-benchmark to run the tests inside WTR, and create a new step running this script that will be executed on the test bots. - Deploy a new bot that runs run-perf-tests on a full-fledged browser like Safari or Epiphany. I wonder what you think is the best option or if there is some option not viable. From my PoV, the option #1 has the advantage of reusing the current infrastructure that collects and draws performance data at https://perf.webkit.org > It might be useful for Mac and other ports to have bots available on > build.webkit.org as well. > Indeed.
signature.asc
Description: OpenPGP digital signature
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev