> On Feb 2, 2016, at 4:56 PM, Carlos Alberto Lopez Perez <clo...@igalia.com> > wrote: > > On 02/02/16 19:58, Ryosuke Niwa wrote: >> On Tue, Feb 2, 2016 at 10:42 AM, Carlos Alberto Lopez Perez >>> But this script seems focused on comparing the performance between >>> different browsers (safari vs chrome vs firefox) rather than in testing >>> and comparing the performance between different revisions of WebKit. >> >> Not at all. It simply supports running benchmark in other browsers. >> >>> Do you think it makes any difference (from the point of view of >>> detecting failures, not from the performance PoV) to run this tests in a >>> full-fledged browser like Safari rather than in WebKitTestRunner ? >> >> Yes. There are many browser features that can significantly impact the >> real world performance. >> > > I'm specifically not asking about performance, but about correctness. > > This discussion was started because Filip said that running JS tests on > a browser catches many failures that are not cached when running the > tests from the terminal. > > So, I'm wondering if running the JS tests on WTR or Safari makes any > difference when catching failures.
I suspect that the browser will catch more failures. > >>> We already have a performance test bot running tests inside WTR. >>> And I see that the current set of tests executed on this bot already >>> includes Speedometer, and that JetStream and Sunspider are skipped on >>> PerformanceTests/Skipped. >>> >>> So I see some options going forward: >>> >>> - Fix the JetStream and Sunspider tests so they can be run as part of >>> the current run-perf-tests script that the performance bots execute. >> >> We should use run-benchmark instead since run-benchmark spits out the >> JSON file that's compatible with run-pref-tests. >> > > I'm a bit lost here. Are you planning to deprecate run-perf-tests with > run-benchmark? What is wrong with run-perf-tests? > >>> - Implement support on the script run-benchmark to run the tests inside >>> WTR, and create a new step running this script that will be executed on >>> the test bots. >> >> I don't see a point in doing this. Why is it desirable to run these >> benchmarks inside WebKitTestRunner? >> > > Less dependencies: WTR (or the MiniBrowser) is something that is > currently built by the bots on each build. > If we want to use Epiphany (for example) for the performance tests, is > another thing we have to take care of building before each run. Not a > big deal, but I wonder if is really worth. > >>> - Deploy a new bot that runs run-perf-tests on a full-fledged browser >>> like Safari or Epiphany. >> >> We should just do this. >> >>> I wonder what you think is the best option or if there is some option >>> not viable. >>> >>> From my PoV, the option #1 has the advantage of reusing the current >>> infrastructure that collects and draws performance data at >>> https://perf.webkit.org >> >> We have an internal instance of the same dashboard to which we're >> reporting results of run-benchmark script. >> > > What about making this public? We will happily contribute with a > GTK+/Linux buildbot for it. > > > Regards. > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev