On 2017-10-12 16:28:33 +1100 (+1100), Ian Wienand wrote: [...] > - logs issues > > Should be behind us. The logs partition ran out of inodes, causing > log upload failures. Pruning jobs should have rectified this. [...]
Just getting caught up now that I have caffeine, but this problem seems to have returned. The pruning passes are still underway (and likely take days to traverse the ~0.8 billion inodes on that 12TB volume so that much is expected), but it's apparent we're saving log files from jobs faster than we can find and delete old ones at this point. I expect our only recourse for now is to further reduce log retention to something like 2 weeks instead of 4 until we can get this under control (if we were already on Zuul v3 we could fold in draft documentation publishing and combine those volumes to buy us some breathing room, but such is life). -- Jeremy Stanley
signature.asc
Description: Digital signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
