On 2014-11-25 07:33:14 -0500 (-0500), Sean Dague wrote: > Right now all icehouse jobs are showing up as NOT_REGISTERED. It's > cross project and current.
This should be solved now for any dsvm icehouse jobs started since 15:00 UTC today. Contributing factors: 1. We restarted Zuul and it doesn't keep a history of Gearman worker registrations over restart, relying on the Jenkins masters to reregister what jobs they know about. 2. The Gearman plug-in for Jenkins only registers jobs for a given worker type when a worker of that type is added to that master. 3. Due to extremely low relative demand for devstack-precise workers (as they only run for stable/icehouse branch and backward-compat jobs), Nodepool was only trying to build one at a time. 4. For some reason the latest devstack-precise snapshot in one of our providers' regions was causing nova boot to consistently fail and discard the instance as being in an error state. 5. Due to the way the Nodepool scheduler is designed, when it is retrying to boot one single node of a given label it retries it in the same provider and region: in this case repeatedly trying to build one single devstack-precise node from a snapshot which was not working in that location. Resolution was ultimately achieved by briefly shuffling the provider order for the devstack-precise label in Nodepool's configuration, causing it to retry in a different region where the boot ultimately succeeded and got attached to a Jenkins master which then registered all jobs requiring devstack-precise nodes into Zuul. -- Jeremy Stanley _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev