On Tue, Feb 2, 2016 at 10:42 AM, Carlos Alberto Lopez Perez <clo...@igalia.com> wrote: > On 02/02/16 00:27, - R. Niwa wrote: >> On Mon, Feb 1, 2016 at 7:32 AM, Carlos Alberto Lopez Perez >> <clo...@igalia.com> wrote: >>> On 31/01/16 05:16, Filip Pizlo wrote: >>>> As my original message said, I was wondering if there was any support >>>> for running some JavaScript tests *in browser*. run-jsc-stress-tests >>>> doesn’t support that because it doesn’t know what a browser is. >>>> >>>> Some tests, like Kraken, Octane, JetStream, and Speedometer, either >>>> require a browser to run (like JetStream and Speedometer) or have >>>> significantly different behavior in the browser than in their >>>> command-line harnesses (like Kraken and Octane). If you did have a >>>> bot that ran these tests in some GTK+ or EFL browser, you’d probably >>>> catch bugs that testing the JSC shell cannot catch. >>>> >>>> -Filip >>> >>> Some questions regarding this JSC in-browser benchmarks: >>> >>> How does the Apple port runs this? Do you use some script that is >>> currently available on the WebKit source tree? Are the buildbots running >>> this tests public? >> >> We've been running those benchmarks with >> http://trac.webkit.org/browser/trunk/Tools/Scripts/run-benchmark >> > > I see. > > But this script seems focused on comparing the performance between > different browsers (safari vs chrome vs firefox) rather than in testing > and comparing the performance between different revisions of WebKit.
Not at all. It simply supports running benchmark in other browsers. > Do you think it makes any difference (from the point of view of > detecting failures, not from the performance PoV) to run this tests in a > full-fledged browser like Safari rather than in WebKitTestRunner ? Yes. There are many browser features that can significantly impact the real world performance. > We already have a performance test bot running tests inside WTR. > And I see that the current set of tests executed on this bot already > includes Speedometer, and that JetStream and Sunspider are skipped on > PerformanceTests/Skipped. > > So I see some options going forward: > > - Fix the JetStream and Sunspider tests so they can be run as part of > the current run-perf-tests script that the performance bots execute. We should use run-benchmark instead since run-benchmark spits out the JSON file that's compatible with run-pref-tests. > - Implement support on the script run-benchmark to run the tests inside > WTR, and create a new step running this script that will be executed on > the test bots. I don't see a point in doing this. Why is it desirable to run these benchmarks inside WebKitTestRunner? > - Deploy a new bot that runs run-perf-tests on a full-fledged browser > like Safari or Epiphany. We should just do this. > I wonder what you think is the best option or if there is some option > not viable. > > From my PoV, the option #1 has the advantage of reusing the current > infrastructure that collects and draws performance data at > https://perf.webkit.org We have an internal instance of the same dashboard to which we're reporting results of run-benchmark script. - R. Niwa _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
