Good points, comments below.
"Ed Morley" <emor...@mozilla.com> wrote in message
news:<mailman.1992.1362404580.24452.dev-platf...@lists.mozilla.org>...
(CCing auto-to...@mozilla.com)
jmaher and jhammel will be able to comment more on the talos specifics,
but few thoughts off the top of my head:
It seems like we're conflating multiple issues here:
1) "[talos] is a separate repo from mc"
2) "[it's a hassle to] test the test to be sure it’s working"
3) "[it's a hassle to get results] populated in graph server"
4) "[we need to] come up with a good way to manage the life cycle of
active perf tests so graph server doesn’t become polluted"
Switching from the talos harness to mochitest doesn't fix #2 (we still
have to test, and I don't see how it magically becomes any easier without
extra work - that could have been applied to talos instead)
I disagree here, very few devs are familiar with the talos framework and
what it takes to get a new test written. Everyone is very familiar with
mochitest and other related test frameworks on mc. I can write a mochitest
to test perf in something simple like scrolling in about an hour. Putting
together a talos scroll test would take much longer. If Talos were on mc it
would help, but integrating into existing test frameworks we have and use on
a regular basis seems like the simplest approach with the least amount of
overhead.
Instead, I think we should be asking:
1) Is the best test framework for performance testing: [a] talos (with
improvements), [b] mochitest (with a significant amount of work to make it
compatible), or [c] a brand new framework?
On [b] there might be a significant amount of work in getting infra pieces
to work maybe (like graph server or whatever we plan to replace it with) but
not in writing a import module that devs would use to post data.
2) Regardless of framework used, would checking it into mozilla-central
improve dev workflow enough to outweigh the downsides (see bug 787200 for
history on that discussion)?
Maybe we might want to keep talos around for "big, important tests". But I
think devs need a way to run perf tests on a smaller scale that doesn't
involve infra changes. I think having this ability would be a big win for
us.
Jim
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
