Re: Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)
Jeremy Katz wrote: On Tue, 2008-05-13 at 23:55 +1200, Nigel Jones wrote: Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge. Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release. We can't do this. We need to keep the tarballs for packages which were released (in any form, be it rawhide, an update, or the actual release -- and probably even updates-testing) for 3 years to comply with the terms of the GPL. Not to mention that removing them would break any ability to go back and try to bisect or find the source of a problem in an automated fashion. And yes, I've actually had to go back to ancient history to do so in the past with some packages :-/ We were just discussing this on #fedora-admin, the other alternative would be (as I mentioned) move them to some sort of publically available archive storage setup, with the Makefile pointing towards that instead of where we normally point. Anyway, I'm just trying to kick the ball about, I'm sure there is a miracle solution but imo at the very least it'd be somewhat nice to unladen cvs-int of some of the burden. - Nigel ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)
On Wed, 2008-05-14 at 00:39 +1200, Nigel Jones wrote: Jeremy Katz wrote: On Tue, 2008-05-13 at 23:55 +1200, Nigel Jones wrote: Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge. Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release. We can't do this. We need to keep the tarballs for packages which were released (in any form, be it rawhide, an update, or the actual release -- and probably even updates-testing) for 3 years to comply with the terms of the GPL. Not to mention that removing them would break any ability to go back and try to bisect or find the source of a problem in an automated fashion. And yes, I've actually had to go back to ancient history to do so in the past with some packages :-/ We were just discussing this on #fedora-admin, the other alternative would be (as I mentioned) move them to some sort of publically available archive storage setup, with the Makefile pointing towards that instead of where we normally point. If we're going to do that, we might as well use that for the tarball repo always and not have it special cased for old vs not. At which point, the easy answer is just to nfs mount that storage as /repo/pkgs :) In fact, maybe that is the easy answer -- maybe we should just allocate some of the space off of nfs1. We're at below 200G of usage for /repo/pkgs -- if we allocated half a terabyte, it wouldn't be that big of a hit to what's available to koji and it would give us quite a bit of room to grow for /repo/pkgs. Jeremy ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)
On Tue, 2008-05-13 at 23:55 +1200, Nigel Jones wrote: Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge. Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release. We can't do this. We need to keep the tarballs for packages which were released (in any form, be it rawhide, an update, or the actual release -- and probably even updates-testing) for 3 years to comply with the terms of the GPL. Not to mention that removing them would break any ability to go back and try to bisect or find the source of a problem in an automated fashion. And yes, I've actually had to go back to ancient history to do so in the past with some packages :-/ Jeremy ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)
Jeremy Katz wrote: On Wed, 2008-05-14 at 00:39 +1200, Nigel Jones wrote: Jeremy Katz wrote: On Tue, 2008-05-13 at 23:55 +1200, Nigel Jones wrote: Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge. Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release. We can't do this. We need to keep the tarballs for packages which were released (in any form, be it rawhide, an update, or the actual release -- and probably even updates-testing) for 3 years to comply with the terms of the GPL. Not to mention that removing them would break any ability to go back and try to bisect or find the source of a problem in an automated fashion. And yes, I've actually had to go back to ancient history to do so in the past with some packages :-/ We were just discussing this on #fedora-admin, the other alternative would be (as I mentioned) move them to some sort of publically available archive storage setup, with the Makefile pointing towards that instead of where we normally point. If we're going to do that, we might as well use that for the tarball repo always and not have it special cased for old vs not. At which point, the easy answer is just to nfs mount that storage as /repo/pkgs :) In fact, maybe that is the easy answer -- maybe we should just allocate some of the space off of nfs1. We're at below 200G of usage for /repo/pkgs -- if we allocated half a terabyte, it wouldn't be that big of a hit to what's available to koji and it would give us quite a bit of room to grow for /repo/pkgs. You know, that's quite sane and I really like that idea. :) - Nigel ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)
Nigel Jones ([EMAIL PROTECTED]) said: Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release. Until we get something like correspondingsource up and running, we don't have a good mechanism for weeding out only the tarballs that correspond to builds that were not/are not shipped. Bill ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)
On Tue, 2008-05-13 at 08:58 -0500, Dennis Gilmore wrote: On Tuesday 13 May 2008, Nigel Jones wrote: Hey guys, I seem to have deleted the nagios notification that I want to mention in particular, but as I noted in SOP/Nagios the %ages noted in the e-mails are what is left, so inode=99% doesn't mean that 99% of the inodes are used, it means 99% are still free. Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge. Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release. snip We already have plans to move the lookaside cache to the netapp. at least that was the last plan i was aware of. Diskspace isn't cheap, so I like delete old tarballs, I also like this option because it's not like they disappear completely, they should be in the src.rpm's already on archive.fp.o and if we accidentally delete 1 or 2 that are still needed, well grab it from src.rpm... This leads on to my second item... xenbuilder2 has run out of diskspace in /, it's down to 32M, thankfully koji has disabled it so it's safe for now, but wouldn't it be nice to throw say a 50GB partition dedicated solely for /var/lib/mock /mnt/build? Yes, yes, I know money, but once again, builds are getting bigger so 'it'd be nice'. xenbuilder1 and hammer2 are the oldest boxes we have in the buildsystem. closely followed by ppc1 xenbuilder2 is also quite an old box now. im cleaning up /var/lib/mock on xenbuilder2 now. failed builds dont get cleaned up automatically. we probably should reap them more often. Buildroots for failed Koji builds do get cleaned up after 4 hours (to give someone time to diagnose the problem if desired). However, xenbuilder2 is also running plague, which never cleans up buildroots as far as I know. ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)
On Tuesday 13 May 2008, Nigel Jones wrote: Hey guys, I seem to have deleted the nagios notification that I want to mention in particular, but as I noted in SOP/Nagios the %ages noted in the e-mails are what is left, so inode=99% doesn't mean that 99% of the inodes are used, it means 99% are still free. Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge. Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release. snip We already have plans to move the lookaside cache to the netapp. at least that was the last plan i was aware of. Diskspace isn't cheap, so I like delete old tarballs, I also like this option because it's not like they disappear completely, they should be in the src.rpm's already on archive.fp.o and if we accidentally delete 1 or 2 that are still needed, well grab it from src.rpm... This leads on to my second item... xenbuilder2 has run out of diskspace in /, it's down to 32M, thankfully koji has disabled it so it's safe for now, but wouldn't it be nice to throw say a 50GB partition dedicated solely for /var/lib/mock /mnt/build? Yes, yes, I know money, but once again, builds are getting bigger so 'it'd be nice'. xenbuilder1 and hammer2 are the oldest boxes we have in the buildsystem. closely followed by ppc1 xenbuilder2 is also quite an old box now. im cleaning up /var/lib/mock on xenbuilder2 now. failed builds dont get cleaned up automatically. we probably should reap them more often. Dennis signature.asc Description: This is a digitally signed message part. ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list