Bug#1020217: S3-backed snapshot implementation on AWS?
On Sun, Sep 24, 2023 at 04:09:31PM -0700, Noah Meyerhans wrote: > > > Could we use the Debian AWS account to host that service? > > > > I would assume that a service like snapshot would be within the scope > > for our AWS usage. Noah? > > It makes sense and I will look into it. Let's not start anything until > we hear definitive confirmation. OK, let's do it. noah
Bug#1020217: S3-backed snapshot implementation on AWS?
Hi, On 24/09/23 at 16:09 -0700, Noah Meyerhans wrote: > On Fri, Sep 22, 2023 at 05:12:21PM +0200, Bastian Blank wrote: > > > Could we use the Debian AWS account to host that service? > > > > I would assume that a service like snapshot would be within the scope > > for our AWS usage. Noah? > > It makes sense and I will look into it. Let's not start anything until > we hear definitive confirmation. Do we have a sense of how much > outgoing traffic the current snapshot service generates? >From #debian-admin: lucas: https://munin.debian.org/debian.org/sallinen.debian.org/ip_193_62_202_27.html and https://munin.debian.org/debian.org/sallinen.debian.org/ip_2001_630_206_4000_1a1a_0_c13e_ca1b.html I think, so average of 35Mbit/sec over the last week. > > However we need to talk about that "one […] VM", because this sounds > > like you intend to use AWS as VM hosting, which it is not. > > > > Please think about this in form of services and there should be at least > > two: > > - the injestor, which can only exist once and writes, and > > - the web frontend, which should be able to exist several times and only > > reads. > > > > So you want to plan with running the multiple web frontends with load > > balancers and maybe even cloudfront. > > I agree that it would be best to design something more cloud-oriented. > However, if there's an existing infrastructure that can be moved as a > "lift & shift" into AWS now, with architectural refactoring happening > later, that's an OK place to start. Yes, that would be the plan I think: start with moving to AWS and replacing the filesystem-backed storage backend to an S3-backed on. Then look at other aspects. Lucas
Bug#1020217: S3-backed snapshot implementation on AWS?
On Fri, Sep 22, 2023 at 05:12:21PM +0200, Bastian Blank wrote: > > Could we use the Debian AWS account to host that service? > > I would assume that a service like snapshot would be within the scope > for our AWS usage. Noah? It makes sense and I will look into it. Let's not start anything until we hear definitive confirmation. Do we have a sense of how much outgoing traffic the current snapshot service generates? > > It would > > require one fairly powerful VM, and a large S3 bucket (approximately > > 150-200 TB). > > 200 TB should be no problem. Agreed. > However we need to talk about that "one […] VM", because this sounds > like you intend to use AWS as VM hosting, which it is not. > > Please think about this in form of services and there should be at least > two: > - the injestor, which can only exist once and writes, and > - the web frontend, which should be able to exist several times and only > reads. > > So you want to plan with running the multiple web frontends with load > balancers and maybe even cloudfront. I agree that it would be best to design something more cloud-oriented. However, if there's an existing infrastructure that can be moved as a "lift & shift" into AWS now, with architectural refactoring happening later, that's an OK place to start. noah
Bug#1020217: S3-backed snapshot implementation on AWS?
Hi Lucas On Fri, Sep 22, 2023 at 08:42:10AM +0200, Lucas Nussbaum wrote: > Could we use the Debian AWS account to host that service? I would assume that a service like snapshot would be within the scope for our AWS usage. Noah? > It would > require one fairly powerful VM, and a large S3 bucket (approximately > 150-200 TB). 200 TB should be no problem. However we need to talk about that "one […] VM", because this sounds like you intend to use AWS as VM hosting, which it is not. Please think about this in form of services and there should be at least two: - the injestor, which can only exist once and writes, and - the web frontend, which should be able to exist several times and only reads. So you want to plan with running the multiple web frontends with load balancers and maybe even cloudfront. Regards, Bastian -- I object to intellect without discipline; I object to power without constructive purpose. -- Spock, "The Squire of Gothos", stardate 2124.5
Bug#1020217: S3-backed snapshot implementation on AWS?
Hi Bastian, I'm playing with the idea of a S3-backed snapshot.d.o implementation (see #1020217). Could we use the Debian AWS account to host that service? It would require one fairly powerful VM, and a large S3 bucket (approximately 150-200 TB). Best, Lucas
