For metrofx we’ve been working on getting omtc and apzc running in the browser. One of the things we need to be able to do is run performance tests that tell us whether or not the work we’re doing is having a positive effect on perf. We currently don’t have automated tests up and running for metrofx and talos is even farther off.
So to work around this I’ve been putting together some basic perf tests I can use to measure performance using the mochitest framework. I’m wondering if this might be a useful answer to our perf tests problems long term. Putting together talos tests is a real pain. You have to write a new test using the talos framework (which is a separate repo from mc), test the test to be sure it’s working, file rel eng bugs on getting it integrated into talos test runs, populated in graph server, and tested via staging to be sure everything is working right. Overall the overhead here seems way too high. Maybe we should consider changing this system so devs can write performance tests that suit their needs that are integrated into our main repo? Basically: 1) rework graphs server to be open ended so that it can accept data from test runs within our normal test frameworks. 2) develop of test module that can be included in tests that allows test writers to post performance data to graph server. 3) come up with a good way to manage the life cycle of active perf tests so graph server doesn’t become polluted. 4) port existing talos tests over to the mochitest framework. 5) drop talos. Curious what people think of this idea. Jim _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
