On 6/14/18 3:50 AM, Emilien Macchi wrote:
TL;DR: gate queue was 25h+, we put all patches from gate on standby, do
not restore/recheck until further announcement.
We recently enabled the containerized undercloud for multinode jobs and
we believe this was a bit premature as the container download process
wasn't optimized so it's not pulling the mirrors for the same containers
multiple times yet.
It caused the job runtime to increase and probably the load on docker.io
<http://docker.io> mirrors hosted by OpenStack Infra to be a bit slower
to provide the same containers multiple times. The time taken to prepare
containers on the undercloud and then for the overcloud caused the jobs
to randomly timeout therefore the gate to fail in a high amount of
times, so we decided to remove all jobs from the gate by abandoning the
patches temporarily (I have them in my browser and will restore when
things are stable again, please do not touch anything).
Steve Baker has been working on a series of patches that optimize the
way we prepare the containers but basically the workflow will be:
- pull containers needed for the undercloud into a local registry, using
infra mirror if available
- deploy the containerized undercloud
- pull containers needed for the overcloud minus the ones already pulled
for the undercloud, using infra mirror if available
- update containers on the overcloud
- deploy the containerized undercloud
Let me also note that it's may be time to introduce jobs dependencies
[0]. Dependencies might somewhat alleviate registries/mirrors DoS
issues, like that one we have currently, by running jobs in batches, and
not firing of all at once.
We still have options to think of. The undercloud deployment takes
longer than standalone, but provides better coverage therefore better
extrapolates (and cuts off) future overcloud failures for the dependent
jobs. Standalone is less stable yet though. The containers update check
may be also an option for the step 1, or step 2, before the remaining
multinode jobs execute.
Making those dependent jobs skipped, in turn, reduces DoS effects caused
to registries and mirrors.
[0]
https://review.openstack.org/#/q/status:open+project:openstack-infra/tripleo-ci+topic:ci_pipelines
With that process, we hope to reduce the runtime of the deployment and
therefore reduce the timeouts in the gate.
To enable it, we need to land in that order:
https://review.openstack.org/#/c/571613/,
https://review.openstack.org/#/c/574485/,
https://review.openstack.org/#/c/571631/ and
https://review.openstack.org/#/c/568403.
In the meantime, we are disabling the containerized undercloud recently
enabled on all scenarios: https://review.openstack.org/#/c/575264/ for
mitigation with the hope to stabilize things until Steve's patches land.
Hopefully, we can merge Steve's work tonight/tomorrow and re-enable the
containerized undercloud on scenarios after checking that we don't have
timeouts and reasonable deployment runtimes.
That's the plan we came with, if you have any question / feedback please
share it.
--
Emilien, Steve and Wes
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Best regards,
Bogdan Dobrelya,
Irc #bogdando
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
