Hey Tom, Where is that script being fired from (a quick grep doesn't find it), or is it a tool people are using?
If it's a tool we'd need to make sure whoever is using it gets a new version to rule it out. - Josh On Tue, Feb 14, 2017 at 7:07 PM, Tom Fifield <t...@openstack.org> wrote: > On 14/02/17 16:06, Joshua Hesketh wrote: > >> Hey, >> >> I've brought the service back up, but have no new clues as to why. >> > > Cheers. > > Going to try: https://review.openstack.org/#/c/433478/ > to see if this script is culprit. > > > - Josh >> >> On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield <t...@openstack.org >> <mailto:t...@openstack.org>> wrote: >> >> On 10/02/17 22:39, Jeremy Stanley wrote: >> >> On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote: >> [...] >> >> Down again, this time with "Network is unreachable". >> >> [...] >> >> I'm not finding any obvious errors on the server nor relevant >> maintenance notices/trouble tickets from the service provider to >> explain this. I do see conspicuous gaps in network traffic volume >> and system load from ~06:45 to ~08:10 UTC according to cacti: >> >> http://cacti.openstack.org/?tree_id=1&leaf_id=156 >> <http://cacti.openstack.org/?tree_id=1&leaf_id=156> >> >> Skipping back through previous days I find some similar gaps >> starting anywhere from 06:30 to 07:00 and ending between 07:00 and >> 08:00 but they don't seem to occur every day and I'm not having >> much >> luck finding a pattern. It _is_ conspicuously close to when >> /etc/cron.daily scripts get fired from the crontab so might >> coincide >> with log rotation/service restarts? The graphs don't show these >> gaps >> correlating with any spikes in CPU, memory or disk activity so it >> doesn't seem to be resource starvation (at least not for any >> common >> resources we're tracking). >> >> >> Indeed. It's down again today during the same timeslot. >> >> Another idea for the cron-based theory: >> >> https://github.com/openstack/uc-recognition/blob/master/tool >> s/get_active_moderator.py >> <https://github.com/openstack/uc-recognition/blob/master/too >> ls/get_active_moderator.py> >> >> loops through the list of Ask OpenStack users via the API on a cron >> running on www.openstack.org <http://www.openstack.org>. Not sure >> when that cron runs, but if it's similar, this could potentially be >> a high-load generator. >> >> >> >> >> Regards, >> >> >> Tom >> >> >> _______________________________________________ >> OpenStack-Infra mailing list >> OpenStack-Infra@lists.openstack.org >> <mailto:OpenStack-Infra@lists.openstack.org> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra> >> >> >> >
_______________________________________________ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra