On Saturday, 31 January 2015 13:08:07 CEST, Inge Wallin wrote:
Well, all of the above and more. Hosting, electricity, networking,
I'm including all of the above as "HW costs" in my proposal. We (KDE) do not have our own datacenter after all.
manual work as the number of physical machines increas
Due to the nature of build jobs which constitute a pretty bursty load, renting VMs sounds like a cost-effective approach for our scale. I do not expect that it would make financial sense for us to procure enough physical HW to cover the peak demand -- hence the VMs. Renting VMs also enables corporate sponsors to offer unused capacity for less money, or even for free.
Another benefit is that bootstrapping a new VM is about executing a single command (and this is not hype; it's how all of the current build VMs in use by Gerrit were launched).
manual work as the complexity increases, and so on.
I would prefer if we stopped using phrases such as "increased complexity", and switched to expressing a specific and measurable difference in costs.
The reason for this request is that "increased complexity" is almost always true, yet people have no way of actually judging what it is going to mean. For example, we have in-house expertise in Jenkins where Ben, Nicolas, Scarlett and Marko understand the stack we use. Switching to anything else will cost us some time to get the respective sysadmins up to speed to the new tools. I do not question this.
However, chances are that at the same time adopting a new system could provide significant savings in future. For example, if a new system allowed easier updates to the build structure, it is reasonable to have a wiki page with instructions and to let developers propose changes for their projects. That way, sysadmin time could be freed for other tasks. So even though there is some initial investment, the long-term operation can easily get less demanding, and cheaper.
In fact, a better tooling could attract more people. "Hey, this is a nice system which provides almost exactly what I need; can I improve it?" It also presents an opportunity to clean some of the cruft accumulated over years, thereby reducing the barrier of entry for new contributors and improving the bus factor.
Finally, there is also that aspect of providing a better service to our developers. Running infrastructure costs something, yet we do it. IMHO, we should strive for offering the best service that we can afford with the resources we have today, and have a contingency plan for future.
Right now, we have access to some nice CPUs for free. If it proves useful and we get used to it and if the resources stop being free at some future point and if we cannot obtain a free alternative from some other sponsor, then we should take a look on how much money it would cost us on a comercial basis at that point. If the price is too big, then the service is *not* that useful for us after all, and we will live without it just fine. If, however, we decide that the cost is worth the price and no free/cheaper altenrative exists, then we should go and pay. There is no vendor lock-in, and it is trivial to migrate to another provider. I just don't see why we should be actively prevented from using resources which we have today just because we might have to ask somewhere else in future. Turning off pre-merge CI would get us back to the current level of resource utilization immediately.
That's what this proposal is all about, and please keep in mind that it's something which is already deployed and tested. It isn't an out-of-thin-blue proposal with uncertain cost of deployment.
Everything can be automated in theory but in practice there is never a fully automatic system.
Yes, which is why it's important to looks at the architecture and to spell out what our expectations of maintenance effort are. It might be reasonable to e.g. compare them with the current effort which goes into keeping Jenkins alive and modernizing it.
Keeping the current state has its very real cost, too. With kind regards, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
