Hi all, we had an outage of Wikitech and Labsconsole / Labs.
Apache on virt0 was broken. It was reported by S Page on IRC, the symptons were wikitech and labsconsole being down via HTTP(S) but pingable. We also had Icinga monitoring telling us about it. I went to virt0 and saw Apache running, shortly after restarted it and that fixed the issue. The root cause i found to be a bug in Phusion Passenger module that is used on virt0, as opposed to cluster Apache. from /usr/lib/phusion_passenger/passenger-spawn-server:61 *** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 23592): I made a copy of the full log and for now it is on virt0 in /root/apache2_error_log_20131219.log if you would like to see it all. So far i just blame https://www.phusionpassenger.com/ respectively libapache2-mod-passenger and didn't take any further action. I did tell the labs IRC channel but did not post on labs-l. Daniel -- Daniel Zahn <dz...@wikimedia.org> Operations Engineer _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l