On 07/28/2013 08:02 PM, Monty Taylor wrote:
Hey all!

There is currently an issue with which is causing a very high failure
rate in the gate. From IRC:

18:32:19          clarkb | the grenade failures seem to get very
consistent in the gate at 2013-0-27 1552UTC
18:32:27          clarkb | before that the success rate is much higher
18:34:53          clarkb | *2013-07-27
18:40:01          clarkb | https://review.openstack.org/#/c/38810/ was
the last change to pass grenade when it was semi consistently passing
18:41:31          clarkb | 38587 and 28082 seem like strong candidates
for the breakage

The working hypothesis is that since the grenade gate is assymetrical
(it consumes grizzly and trunk but only gates trunk) that a change to
grizzly went in that broke something for trunk. Obviously this is
something we want to avoid - but since this is our first time gating on
upgrade patterns in this way, it's also probably a good chance for us to
learn about the process of doing that.

In any case, although I'm sure dtroyer and sdague will take a look as
soon as they are online, it's unlikely that anything is going to land
until this is sorted- so I'm sure they'd appreciate any help from anyone
who can look in to the actual issue.

I won't be able to get to trying code until tomorrow morning, however the mostly likely culprit line I'm seeing in the logs is this - http://logs-dev.openstack.org/51/38951/2/check/gate-grenade-devstack-vm/22154/logs/new/screen-c-sch.txt.gz#2013-07-28%2006%3A27%3A34.974

Cinder isn't able to schedule volumes, which is bad. How we got to this stage of bad post upgrade is unknown to me.

The other thing that is suspicious is this - http://logs-dev.openstack.org/51/38951/2/check/gate-grenade-devstack-vm/22154/logs/new/screen-c-vol.txt.gz#2013-07-28%2006%3A26%3A36.466

Especially given that the last change that passed the gate was adding jsonschema to the tempest requirements list. Maybe this is all just a crazy requirements unwind?

Anyway, help appreciated on debugging. This is actually catching a real problem with cinder, which is what it was designed to do. How we got to the real problem is however kind of up in the air.

        -Sean

--
Sean Dague
http://dague.net

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to