Tl;dr tripleo ci is back up and running, see below for more
On 21/03/15 01:41, Dan Prince wrote:
Short version:
The RH1 CI region has been down since yesterday afternoon.
We have a misbehaving switch and have file a support ticket with the
vendor to troubleshoot things further. We hope to know more this
weekend, or Monday at the latest.
Long version:
Yesterday afternoon we started seeing issues in scheduling jobs on the
RH1 CI cloud. We haven't made any OpenStack configuration changes
recently, and things have been quite stable for some time now (our
uptime was 365 days on the controller).
Initially we found a misconfigured Keystone URL which was preventing
some diagnostic queries via OS clients external to the rack. This
setting hadn't been recently changed however and didn't seem to bother
nodepool before so I don't think it is the cause of the outage...
MySQL also got a bounce. It seemed happy enough after a restart as well.
After fixing the keystone setting and bouncing MySQL instances appears
to go ACTIVE but we were still having connectivity issues getting
floating IPs and DHCP working on overcloud instances. After a good bit
of debugging we started looking at the switches. Turns out one of them
has a high CPU usuage (above the warning threshold) and MAC address are
also unstable (ports are moving around).
Until this is resolved RH1 is unavailable to host jobs CI jobs. Will
post back here with an update once we have more information.
RH1 has been running as expected since last Thursday afternoon which
means the cloud was down for almost a week, I'm left not entirely sure
what some problems were, at various times during the week we tried a
number of different interventions which may have caused (or exposed)
some of our problems, e.g.
at one stage we restarted openvswitch in an attempt to ensure nothing
had gone wrong with our ovs tunnels, around the same time (and possible
caused by the restart), we started getting progressively worse
connections to some of our servers. With lots of entries like this on
our bastion server
Mar 20 13:22:49 host01-rack01 kernel: bond0.5: received packet with own
address as source address
Not linking the restart with the looping packets message and instead
thinking we may have a problem with the switch we put in a call with our
switch vendor.
Continuing to chase down a problem on our own servers we noticed that
tcpdump was reporting at times about 100,000 ARP packets per second
(sometimes more).
Various interventions stopped the excess broadcast traffic e.g.
Shutting down most of the compute nodes stopped the excess traffic,
but the problem wasn't linked to any one particular compute node
Running the tripleo os-refresh-config script on each compute node
stopped the excess traffic
But restarting the controller node caused the excess traffic to return
Eventually we got the cloud running without the flood of broadcast
traffic, with a small number of compute nodes, but instances still
weren't getting IP address, with nova and neutron in debug mode we saw
an error where nova failing to mount the qcow image (iirc it was
attempting to resize the image).
Unable to figure out why this was working in the past but now isn't we
redeployed this single compute node using the original image that was
used (over a year ago), instances on this compute node we're booting but
failing to get an IP address, we noticed this was because of a
difference between the time on the controller when compared to the
compute node. After resetting the time, now instances were booting and
networking was working as expected (this was now Wednesday evening).
Looking back at the error while mounting the qcow image, I believe this
was a red herring, it looks like this problem was always present on our
system but we didn't have scary looking tracebacks in the logs until we
switched to debug mode.
Now pretty confident we can get back to a running system by starting up
all the compute nodes again and ensuring the os-refresh-config scripts
were run then ensuring the times were all set on each host properly we
decided to remove any entropy the may have built up while debugging
problems on each computes node so we redeployed all of our compute nodes
from scratch. This all went as expected but was a little time consuming
as we spent time to verify each step as we went along, the steps went
something like this
o with the exception of the overcloud controller, "nova delete" all of
the hosts on the undercloud (31 hosts)
o we now have a problem, in tripleo the controller and compute nodes are
tied together in a single heat template, so we need the heat template
that was used a year ago to deploy the whole overcloud along with the
parameters that were passed into it, we had actually done this before
when adding new compute nodes to the cloud so it wasn't new territory.
o Use "heat template-show ci-overcloud" to get the original heat
template (a json version) that was used and remove anything referring to
the controller
o Edit a version of devtest overcloud to use this template and to
skip various other steps
o Our overcloud had 3 VM instances used by CI that now needed to be
replaced, each needed specific IP addresses and to be on both the
default and test network
o squid - nova boot the image we had for this
o bandersnatch - nova boot the image we had for this
o te-broker - we didn't have an image for this, we booted a vanilla
Fedora image and manually installed geard
o testenv hosts - we also redeployed the test env hosts
o set the times on all of the hosts so that they are in sync
Its thursday lunchtime and our cloud is now back up and running, at this
point we remove the IP tables rule preventing nodepool from talking to
our cloud
We still have problems, regressions have been committed to various non
tripleo repositories causing our tests to fail in various ways (4
regressions in total)
1. Fedora instances started by nodepool were failing to boot, we
eventually tracked this down to an update to the scripts nodpool uses to
build this image, this update was only needed for a specific cloud, so
our fix here was to only run it on that cloud [1]
2. A neutron regression [2]
3. A keystone regression [3]
4. A horizon regression[4]
[1] https://review.openstack.org/#/c/168196/
[2] https://bugs.launchpad.net/tripleo/+bug/1437116
[3] https://bugs.launchpad.net/tripleo/+bug/1437032
[4] https://bugs.launchpad.net/horizon/+bug/1438133
With these four problems now fixed,reverted or temporarily worked around
the tripleo CI is back running and jobs are passing.
I'm pretty confident we'll never be sure what the initial problem was
but here is what I believe
At the start of all this our cloud had only a single symptomatic problem
being caused by time drifting on compute nodes. This was causing a high
percentage of our instances to fail to boot (depending on when various
agents last reported back to the controller), this problem was probably
getting progressively worse over the last while but nodepool was
handling it by deleting instances and restarting new ones until in got
an instance that worked (infra may be able to verify this and if the
situation has improved).
The change to the nodepool script which broke our fedora image caused us
to start looking at the cloud for problems, while debugging the cloud
our test instances we're not getting IP addresses (due to time drift)
which set us down the path of a real issue but not the same issue that
prompted use to start looking for problems in the first place.
All of the other problems were either caused by us trying to debug or
fix the cloud or were red herrings.
This cloud was deployed using tripleo over a year ago, at the time ntp
wasn't part of the default install, it is now installed by default. At
some stage in the future when we redeploy it using ironic (its currently
nova-bm) we should ensure it is used to avoid hitting this again. Also
when redeploying we should take advantage of some monitoring that is now
part of tripleo or add some in places where it is lacking.
The rh1 cloud has been up and running for just over a year without any
major problems/outages (at least that we know of), given that its run by
developers in the tripleo community, I think this is a reasonable time
between outages, although I would hope we could debug and fix any future
problems if they arise, in a shorter time frame.
Feel free to ask questions if you want me to elaborate on anything, I
hope I didn't ramble too much,
Derek.
Dan
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
