Re: Lot of snapshots in AWS Sydney region
On 7/12/23 19:03, Miroslav Suchý wrote: > Dne 13. 07. 23 v 0:26 Dusty Mabe napsal(a): >> If you say it's an issue then hopefully we can give this some priority and >> get to it soon. >> >> Removing the "old ones" isn't that easy to do. Our production AMIs and >> development AMIs are all mixed together so it would be hard to come up with >> a criteria without implementing the garbage collection I linked to above. > > But AWS will not allow you to delete snapshot that is associated with AMI > (you said). So we can delete everything older > than one year. And these that will errors are these that we still want to > keep. So we will just ignore the errors. > Indeed. Thanks for clarifying. Yes that should be OK I think. though it would be good to test your process against a few snapshots that back AMIs we know we don't care about first to make sure it fails. I tried in the web interface and it wouldn't let me so I'm pretty sure that's the case. Dusty ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Lot of snapshots in AWS Sydney region
Dne 13. 07. 23 v 0:26 Dusty Mabe napsal(a): If you say it's an issue then hopefully we can give this some priority and get to it soon. Removing the "old ones" isn't that easy to do. Our production AMIs and development AMIs are all mixed together so it would be hard to come up with a criteria without implementing the garbage collection I linked to above. But AWS will not allow you to delete snapshot that is associated with AMI (you said). So we can delete everything older than one year. And these that will errors are these that we still want to keep. So we will just ignore the errors. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Lot of snapshots in AWS Sydney region
On 7/12/23 16:17, Miroslav Suchý wrote: > Dne 11. 07. 23 v 15:53 Dusty Mabe napsal(a): >> Apologies for not responding sooner. Actually for some reason this is the >> first email I've seen >> in the thread so maybe I need to check my spam filters. Either way, >> apologies. >> >> The reason you are seeing snapshots but no volumes is because these >> snapshots are used as backing >> storage for AMIs. If the snapshot is still associated with an AMI AWS won't >> let you delete it. >> >> On the Fedora CoreOS side we need to implement garbage collection so that we >> delete all AMIs and >> snapshots from our development streams. For our production streams we'll >> probably take a more >> conservative approach to garbage collection, but we'll need to start garbage >> collecting those too. >> >> For Fedora Cloud, that working group will also need to look at their >> processes and implement garbage >> collection too. It could either be a separate process or it could be working >> with you to set a >> policy directly in AWS to clean up after some time. >> >> For Fedora CoreOS we'd like to implement the GC outside of AWS since we'd >> like to have the same GC >> policy for all clouds we create resources in. > > Can you create issue for each of the case. So it does not get lost. For Fedora CoreOS we have this issue to track: https://github.com/coreos/fedora-coreos-tracker/issues/99 For the cloud working group we'd need to reach out to them to see how they want to proceed. > >> Does this make sense? > > Sure. Do you expect this soonish? Or should I manually remove the old ones > now? If you say it's an issue then hopefully we can give this some priority and get to it soon. Removing the "old ones" isn't that easy to do. Our production AMIs and development AMIs are all mixed together so it would be hard to come up with a criteria without implementing the garbage collection I linked to above. Dusty ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Lot of snapshots in AWS Sydney region
Dne 11. 07. 23 v 15:53 Dusty Mabe napsal(a): Apologies for not responding sooner. Actually for some reason this is the first email I've seen in the thread so maybe I need to check my spam filters. Either way, apologies. The reason you are seeing snapshots but no volumes is because these snapshots are used as backing storage for AMIs. If the snapshot is still associated with an AMI AWS won't let you delete it. On the Fedora CoreOS side we need to implement garbage collection so that we delete all AMIs and snapshots from our development streams. For our production streams we'll probably take a more conservative approach to garbage collection, but we'll need to start garbage collecting those too. For Fedora Cloud, that working group will also need to look at their processes and implement garbage collection too. It could either be a separate process or it could be working with you to set a policy directly in AWS to clean up after some time. For Fedora CoreOS we'd like to implement the GC outside of AWS since we'd like to have the same GC policy for all clouds we create resources in. Can you create issue for each of the case. So it does not get lost. Does this make sense? Sure. Do you expect this soonish? Or should I manually remove the old ones now? -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Lot of snapshots in AWS Sydney region
On 7/11/23 04:21, Miroslav Suchý wrote: > Dne 05. 07. 23 v 21:17 Kevin Fenzi napsal(a): >> On Mon, Jul 03, 2023 at 04:02:02PM +0200, Fabian Arrotin wrote: >>> On 03/07/2023 06:39, Miroslav Suchý wrote: I was going through our spending in AWS and I found that we spent a lot in Sydney region (?). In details most of the bill there is because of stored snapshots. There is 7508 of them dated to back to 2013. For volumes that does not exists any more. >> Yeah, lets get the Fedora coreos folks to look, it seems like those are >> their images. Perhaps they are making them, but not cleaning up old ones >> in that region? >> > > It has been week with no response (I know holiday season...). I will give it > one more week. If no one raise a voice, I > will create Recycle-bin rule that will automatically delete **ALL** volume > snapshots older than one year. In ALL AWS > regions where we have some snapshots. I will work on that next Monday. > > If you need to preserve some snapshot longer than one year, please let me > know. Apologies for not responding sooner. Actually for some reason this is the first email I've seen in the thread so maybe I need to check my spam filters. Either way, apologies. The reason you are seeing snapshots but no volumes is because these snapshots are used as backing storage for AMIs. If the snapshot is still associated with an AMI AWS won't let you delete it. On the Fedora CoreOS side we need to implement garbage collection so that we delete all AMIs and snapshots from our development streams. For our production streams we'll probably take a more conservative approach to garbage collection, but we'll need to start garbage collecting those too. For Fedora Cloud, that working group will also need to look at their processes and implement garbage collection too. It could either be a separate process or it could be working with you to set a policy directly in AWS to clean up after some time. For Fedora CoreOS we'd like to implement the GC outside of AWS since we'd like to have the same GC policy for all clouds we create resources in. Does this make sense? ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Lot of snapshots in AWS Sydney region
Dne 05. 07. 23 v 21:17 Kevin Fenzi napsal(a): On Mon, Jul 03, 2023 at 04:02:02PM +0200, Fabian Arrotin wrote: On 03/07/2023 06:39, Miroslav Suchý wrote: I was going through our spending in AWS and I found that we spent a lot in Sydney region (?). In details most of the bill there is because of stored snapshots. There is 7508 of them dated to back to 2013. For volumes that does not exists any more. Yeah, lets get the Fedora coreos folks to look, it seems like those are their images. Perhaps they are making them, but not cleaning up old ones in that region? It has been week with no response (I know holiday season...). I will give it one more week. If no one raise a voice, I will create Recycle-bin rule that will automatically delete **ALL** volume snapshots older than one year. In ALL AWS regions where we have some snapshots. I will work on that next Monday. If you need to preserve some snapshot longer than one year, please let me know. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
