With all new deployments at Blackbaud, we have been moving towards the HONEYBADGER model that you described. Each engineering team is responsible for the micro services that they build, including monitoring and support. I run the platform engineering team (could also be called a tools or devops team). We build the underlying platform that everything runs on (AWS, Cloud Foundry, CI/CD tools, etc). We try to let the engineering teams do everything, and when we come across something they can’t do, say because of security or access issues, my team’s job is to support the team immediately to unblock them but then to build a tool that will allow them to do it themselves int he future.
I don’t think that you need separation for highly repeatable tasks, that’s what automation is for. Scaling isn’t a problem for us at this point. Since most features are deployed as individual micro services, the concept of done for a particular feature is much more clearly defined. Once it hits full feature state, it goes into maintenance mode and is assigned a maintainer and any new functionality that is added is added by the team that needs the new functionality. The biggest challenge that we have had in this model is with our customer support teams. With our current generation products, they have a traditional “Ops Team” that they can call to engage The new model is a little more complicated. Our goal on this front is to build our new services such that they are self healing and that our engineering teams already know whats going on and have updated our incident channel in slack before a customer ever calls in, thus eliminating the need for support to reach out to anybody. Anyway, I’m happy to share more if your interested. On 4/24/16, 10:19 PM, "[email protected] on behalf of chris hardy" <[email protected] on behalf of [email protected]> wrote: >Matt, what did you end up coming up with? > >we're moving toward this, but it's not an anarchy. We have a devops team that >supports feature teams, as needed, in writing the chef scripts for their >feature. We designed our infrastructure tooling for this purpose. Sometimes >we provide a fair amount of support, but one big benefit is that we understand >the requirements from the feature team a lot faster when they're responsible >for their own build tooling and collaboration is good so we're all wasting >less time speculating about why this or that doesn't work as expected out of >the box. > >> On Feb 15, 2016, at 2:25 PM, Matt Disney <[email protected]> wrote: >> >> Hi everyone, >> >> I'm looking for experiences of people that have the "you build it, you run >> it" approach (hereafter referred to as "HONEYBADGER") and how you scale that >> up to building/supporting more projects over time. >> >> I've never worked in a large-scale web/services shop (largest IT/product >> division I've been in is 250), and I'm puzzled by how to make the >> HONEYBADGER approach work over time as you continue to add customers, >> services, and complexity in general. We do govern the growth of those things >> but we're intentionally going after emerging research/products, so the >> increase in complexity is inevitable. In some ways, the expense of >> supporting a service/product goes down over time (e.g. you work out the >> problems, you streamline things, etc). But it's also the case that adding >> new features to a product/service and growth in usage/popularity can offset >> those cost/time-savings from quality and efficiency. >> >> Furthermore, to scale HONEYBADGER up, it would seem you need some kind of >> separation for at least highly repeatable tasks. Otherwise, as the people >> that built the N-1 thing continue on to build the current (N) thing and the >> N+1 thing, while still supporting the N-1 thing, it seems you'd run out of >> people to build new stuff. >> >> So how does this work in large shops? Is HONEYBADGER really just about >> software stacks? Do the builders really do customer support? Do you do this >> for IT operations or just products that directly generate revenue? >> >> Thanks, >> Matt >> _______________________________________________ >> Discuss mailing list >> [email protected] >> https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss >> This list provided by the League of Professional System Administrators >> http://lopsa.org/ > >_______________________________________________ >Discuss mailing list >[email protected] >https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss >This list provided by the League of Professional System Administrators > http://lopsa.org/ _______________________________________________ Discuss mailing list [email protected] https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
