> On Feb 9, 2017, at 4:32 PM, Greg Stein <[email protected]> wrote:
>
>
> I also believe we had to pay for that box, and it wasn't cheap.
I had it in my head that it was from the Apple hardware donation that
happened a while back, but that was a long time ago. (So long ago that it
appears to have been wiped from the ASF website.)
> Today, our preferred model for non-Ubuntu boxes is to have other people
> own/run/manage those buildbots and hook them into our build master.
Yup. That's how the PowerPC got added. Thus why I say it's better for
projects to start digging into corporate sponsorship. FWIW: It was not a smooth
process, but there were issues on both the donate-ees and the donate-ers.
> Apache Hadoop is worse for it. As Gavin has noted, just in the past year,
> we've increased our build farm dramatically. I believe the ASF is better
> for it.
I guess I'm viewing this from a different and very selfish perspective.
Prior to your time, a lot of effort was spent by the Hadoop project on reducing
the build infrastructure footprint because the project itself was getting
bogged down, build failures (esp after 2 executors magically appeared on the
nodes), etc, etc. We moved all/most of our builds to be docker-ized to also
minimize the impact on the infra team. End result was significant savings all
around. Many jobs were shutdown, slots were opened up, builds became more
reliable, Yetus was born to share our experiences, etc, etc. The ecosystem of
projects that were running on those nodes also benefitted since Hadoop was
usually the #1 or #2 user. That said: last I checked, there have been no new
build machines added to the Hadoop pool. In fact, we usually run at a deficit
because machines are always down. (right now, we're at something like 60%
capacity: H2,H3,H7,H8,H9 are all offline). So while the build farm may have
increased for the ASF overall, Hadoop is not benefitting from a hardware
increase. We've benefitted from fixing our own build jobs and from the infra
team's work on Jenkins itself.
> Today, if a machine goes down,
> we can spin it back up in an hour or two due to the consistency.
Great, so I should expect for all of our nodes to come back up quickly
then, right? ;)
> I do sympathize that our service reduction is painful. But I hope you can
> understand where the Foundation (and its Infra team) is coming from. We
> have vastly more projects to support today, meaning more uniformity is
> required.
It is what it is. I'm not angry (anymore). While I do really
appreciate a lot of what the infra team goes through (I'm much more of an ops
person than a dev person), it just needs to be pointed out that there are
impacts to teams and that everything isn't completely 100% positive. I've
resigned myself to the fact that it's better for my own sanity to donate my
time to other things than the ASF. Most of that is related to how the Apache
Hadoop project itself is run, but there is a portion of that related to getting
burned by the infra team's changes. I think things are improving, but I'm still
very much in a wait-and-see mode.
Thanks.
