Sorry, [1] meant to reference https://issues.jenkins-ci.org/browse/JENKINS-37984 .
On Sun, Nov 25, 2018 at 5:41 PM kellen sunderland < kellen.sunderl...@gmail.com> wrote: > Marco and I ran into another urgent issue over the weekend that was > causing builds to fail. This issue was unrelated to any feature > development work, or other CI fixes applied recently, but it did require > quite a bit of work from Marco (and a little from me) to fix. > > We spent enough time on the problem that it caused us to take a step back > and consider how we could both fix issues in CI and support the 1.4 release > with the least impact possible on MXNet devs. Marco had planned to make a > significant change to the CI to fix a long-standing Jenkins error [1], but > we feel that most developers would prioritize having a stable build > environment for the next few weeks over having this fix in place. > > To properly introduce a new CI system the intent was to do a gradual > blue/green roll out of the fix. To manage this rollout would have taken > operational effort and double compute load as we run systems in parallel. > This risks outages due to scaling limits, and we’d rather make this change > during a period of low-developer activity, i.e. shortly after the 1.4 > release. > > This means that from now until the 1.4 release, in order to reduce > complexity MXNet developers should only see a single Jenkins verification > check, and a single Travis check. > >
