On Mon, Aug 18, 2014 at 4:35 PM, Maryana Pinchuk <[email protected]> wrote:
> Greetings, QAers, > > I'm not entirely sure who the point person for Beta Labs is currently, but > I wanted to make sure you guys are aware that there have been a lot of > issues (i.e., partial or total outages) with this environment in the past > 2-3 weeks, most likely due to ongoing HHVM work. Unfortunately, because > several teams at WMF rely quite heavily on Beta Labs – such as Mobile Web > for testing new user-facing features before they go live in production, and > Design/UX for running remote and in-person user tests – this is not so > great. > I've started talking to people informally about the current situation with beta labs. TLDR: it is FANTASTIC that beta is so valuable to so many teams and people; but it is difficult to support so many projects on a single beta labs cluster. Originally beta labs was only to be a target for automated browser tests. We needed a shared test environment(*) where "works on my machine" was not an option(**). Beta labs unexpectedly and organically grew into something more important than just that. Then we had a policy of using beta only for software already in production. We broke that policy with CirrusSearch, followed by Flow, and now by testing HHVM in beta labs. I am starting to think we should have at least one more shared cluster test environment running the master branch of code, i.e. a "beta2" test environment. I would like one for modeling the production cluster (like the original beta concept) and a different one for working on system-wide, cross-cutting changes (like CirrusSearch, Flow, HHVM, etc.) I could see having yet another "beta3" env for working with code not yet merged to master. And even the original beta labs could probably use more CPU/memory/etc than it has now. Thanks for bringing this up Maryana, I think the current and future state of beta labs is something that needs scrutiny sooner rather than later. -Chris (*) Just a note, shared test environments like beta labs are out of fashion right now, I remain pleased that ours is so valuable. We should probably blog that somewhere. (**) We should also take a serious look at what is and is not possible with beta and with vagrant, and where to draw the line between a shared test env like beta and shared dev envs like vagrant images. > For example, the outage during Wikimania was particularly bad because > Abbey was hosting a UX testing table during the conference that had > scripted user tests all pointing to Beta Labs, so she was unable to run any > of those tests and lost out on valuable user testing time. > > To avoid situations like this in the future, is there a way for teams who > use Beta Labs for testing to stay in closer sync with its maintainers? I > realize y'all aren't mind-readers ;) and there will of course be unexpected > issues that crop up from time to time. But if there are likely to be more > major breaking changes to the infrastructure while you continue working on > HHVM, it would be great to get an advanced heads-up so we can plan > accordingly. And when unexpected outages do occur, it'd also be good to > know who to report them to and check in on progress with, because I'm not > sure the current strategy of whining and hoping it'll fix itself is > working/sane/scalable :) > > I don't know how this would work ideally (regular updates to a > new/existing mailing list? dropping in on a new/existing IRC channel? a > wiki page? some combo of all three?) but am open to any & all ideas! And as > a start, it would be good if those of you currently tasked with maintaining > Beta Labs could give a virtual wave – I *think* I know who you are, but > Abbey's new and almost certainly doesn't ;) > > -- > Maryana Pinchuk > Product Manager, Wikimedia Foundation > wikimediafoundation.org > > _______________________________________________ > QA mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/qa > >
_______________________________________________ QA mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/qa
