Re: [OpenStack-Infra] [all] Unplanned project infrastructure reboots

2014-09-28 Thread Jeremy Stanley
On 2014-09-28 14:44:20 + (+), Jeremy Stanley wrote:
[...]
> I will follow up again in roughly 12 hours once I'm reasonably
> confident the maintenance activity poses no further risk to these
> systems and have given then a cursory health check to make sure
> nothing's seriously broken prior to heading into the work week.

As predicted, we came through mostly unscathed. At this point
everything looks like it should be working normally, with the
exception of a couple non-critical informational systems we're
continuing to troubleshoot (graphite and logstash/elasticsearch). If
you notice any other systems having issues, please let the project
infrastructure team know in #openstack-infra on freenode.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra


[OpenStack-Infra] [all] Unplanned project infrastructure reboots

2014-09-28 Thread Jeremy Stanley
The OpenStack project infrastructure servers which provide the
majority of our developer and community resources will be down for
brief periods throughout the day while the service provider donating
the resources to run them restarts the underlying Nova compute nodes
over the course of a day-long emergency maintenance window. All our
systems are designed to tolerate random reboots and getting
restarted in any order, but there is still a risk of prolonged
impact (particularly due to unforeseen consequences of interrupting
interactions between interdependent systems).

I will follow up again in roughly 12 hours once I'm reasonably
confident the maintenance activity poses no further risk to these
systems and have given then a cursory health check to make sure
nothing's seriously broken prior to heading into the work week.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra