Re: Amazon EC2 EBS Volume for Maven central repository
the problem for a public dataset it's that AFAIK they are static while the repo keeps changing On Mon, Dec 21, 2009 at 12:55 PM, Mark Diggory wrote: > FYI > > Initiating a thread here to see its effect. > > http://developer.amazonwebservices.com/connect/thread.jspa?messageID=158762𦰪 > > Cheers, > Mark > > On Mon, Dec 21, 2009 at 10:19 AM, Carlos Sanchez wrote: > >> you can set s3 buckets where "requester pays" >> >> >> http://developer.amazonwebservices.com/connect/message.jspa?messageID=123715 >> >> On Mon, Dec 21, 2009 at 9:30 AM, Brian Fox wrote: >> > On Mon, Dec 21, 2009 at 12:09 AM, Mark Diggory >> wrote: >> >> I imagine there have to be a number of projects/companies out there >> >> using Maven artifacts and incurring bandwidth costs to build systems. >> >> Atlassian seems to be recommending the practice to its Bamboo users... >> > >> > >> > They should just have a repo manager running in their ec2 grid anyway >> > to reduce bandwidth. >> > >> > We looked into putting central into s3 but then you loose all control >> > over the access and we would get killed by the scrapers constantly >> > downloading everything because it's "free". >> > - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
FYI Initiating a thread here to see its effect. http://developer.amazonwebservices.com/connect/thread.jspa?messageID=158762𦰪 Cheers, Mark On Mon, Dec 21, 2009 at 10:19 AM, Carlos Sanchez wrote: > you can set s3 buckets where "requester pays" > > > http://developer.amazonwebservices.com/connect/message.jspa?messageID=123715 > > On Mon, Dec 21, 2009 at 9:30 AM, Brian Fox wrote: > > On Mon, Dec 21, 2009 at 12:09 AM, Mark Diggory > wrote: > >> I imagine there have to be a number of projects/companies out there > >> using Maven artifacts and incurring bandwidth costs to build systems. > >> Atlassian seems to be recommending the practice to its Bamboo users... > > > > > > They should just have a repo manager running in their ec2 grid anyway > > to reduce bandwidth. > > > > We looked into putting central into s3 but then you loose all control > > over the access and we would get killed by the scrapers constantly > > downloading everything because it's "free". >
Re: Amazon EC2 EBS Volume for Maven central repository
you can set s3 buckets where "requester pays" http://developer.amazonwebservices.com/connect/message.jspa?messageID=123715 On Mon, Dec 21, 2009 at 9:30 AM, Brian Fox wrote: > On Mon, Dec 21, 2009 at 12:09 AM, Mark Diggory wrote: >> I imagine there have to be a number of projects/companies out there >> using Maven artifacts and incurring bandwidth costs to build systems. >> Atlassian seems to be recommending the practice to its Bamboo users... > > > They should just have a repo manager running in their ec2 grid anyway > to reduce bandwidth. > > We looked into putting central into s3 but then you loose all control > over the access and we would get killed by the scrapers constantly > downloading everything because it's "free". > > - > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > For additional commands, e-mail: users-h...@maven.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
On Mon, Dec 21, 2009 at 12:09 AM, Mark Diggory wrote: > I imagine there have to be a number of projects/companies out there > using Maven artifacts and incurring bandwidth costs to build systems. > Atlassian seems to be recommending the practice to its Bamboo users... They should just have a repo manager running in their ec2 grid anyway to reduce bandwidth. We looked into putting central into s3 but then you loose all control over the access and we would get killed by the scrapers constantly downloading everything because it's "free". - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
Ok, so, from the EBS web page...we can speculate... As an example, a medium sized website database might be 100 GB in size and > expect to average 100 I/Os per second over the course of a month. This would > translate to $10 per month in storage costs (100 GB x $0.10/month), and > approximately $26 per month in request costs (~2.6 million seconds/month x > 100 I/O per second * $0.10 per million I/O). > We might seek out the Freebase folks and inquire for the what their costs are like to provide an EBS dataset. Mark On Mon, Dec 21, 2009 at 12:43 AM, Wayne Fay wrote: >> What's the current storage requirements for the central repository at this time? >> > > Per Jarvana, Central is around 100gb, as of mid October 2009: > http://www.jarvana.com/jarvana/info/repository_statistics > > Wayne > > On Sun, Dec 20, 2009 at 9:09 PM, Mark Diggory wrote: > I imagine there have to be a number of projects/companies out there > using Maven artifacts and incurring bandwidth costs to build systems. > Atlassian seems to be recommending the practice to its Bamboo users... > > > http://confluence.atlassian.com/display/BAMBOO/Populating+your+EBS+volume#PopulatingyourEBSvolume-UploadingMaven2repositorydata > > I know its not much, but on large builds and continuous integration > projects would benefit. > > What's the current storage requirements for the central repository at this > time? > > I wonder if the maven central repo could qualify as a public dataset? > http://aws.amazon.com/publicdatasets/ > http://developer.amazonwebservices.com/connect/forum.jspa?forumID=55 > > Mark > > On Sun, Dec 20, 2009 at 8:28 PM, Carlos Sanchez wrote: > > there was some talk about it during ApacheCON but there are some costs > > associated > > > > - S3 storage of the EBS snapshot > > - bandwidth to upload the repo and keep it up to date > > - an EC2 machine to do the rsync and scheduled snapshots of the EBS > volume > > > > There's an option to "sell" it so users end paying the fees > > > > > > On Sun, Dec 20, 2009 at 8:18 PM, Mark Diggory > wrote: > >> Cheers, > >> > >> After experimenting a bit with EC2, It seems it would be ideal if > >> there were an EBS volume that had all the Maven central repository > >> within it. Does anyone know of such a thing? > >> > >> thanks, > >> Mark > >
Re: Amazon EC2 EBS Volume for Maven central repository
> What's the current storage requirements for the central repository at this > time? > Per Jarvana, Central is around 100gb, as of mid October 2009: http://www.jarvana.com/jarvana/info/repository_statistics Wayne - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
I imagine there have to be a number of projects/companies out there using Maven artifacts and incurring bandwidth costs to build systems. Atlassian seems to be recommending the practice to its Bamboo users... http://confluence.atlassian.com/display/BAMBOO/Populating+your+EBS+volume#PopulatingyourEBSvolume-UploadingMaven2repositorydata I know its not much, but on large builds and continuous integration projects would benefit. What's the current storage requirements for the central repository at this time? I wonder if the maven central repo could qualify as a public dataset? http://aws.amazon.com/publicdatasets/ http://developer.amazonwebservices.com/connect/forum.jspa?forumID=55 Mark On Sun, Dec 20, 2009 at 8:28 PM, Carlos Sanchez wrote: > there was some talk about it during ApacheCON but there are some costs > associated > > - S3 storage of the EBS snapshot > - bandwidth to upload the repo and keep it up to date > - an EC2 machine to do the rsync and scheduled snapshots of the EBS volume > > There's an option to "sell" it so users end paying the fees > > > On Sun, Dec 20, 2009 at 8:18 PM, Mark Diggory wrote: >> Cheers, >> >> After experimenting a bit with EC2, It seems it would be ideal if >> there were an EBS volume that had all the Maven central repository >> within it. Does anyone know of such a thing? >> >> thanks, >> Mark - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Amazon EC2 EBS Volume for Maven central repository
there was some talk about it during ApacheCON but there are some costs associated - S3 storage of the EBS snapshot - bandwidth to upload the repo and keep it up to date - an EC2 machine to do the rsync and scheduled snapshots of the EBS volume There's an option to "sell" it so users end paying the fees On Sun, Dec 20, 2009 at 8:18 PM, Mark Diggory wrote: > Cheers, > > After experimenting a bit with EC2, It seems it would be ideal if > there were an EBS volume that had all the Maven central repository > within it. Does anyone know of such a thing? > > thanks, > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > For additional commands, e-mail: users-h...@maven.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Amazon EC2 EBS Volume for Maven central repository
Cheers, After experimenting a bit with EC2, It seems it would be ideal if there were an EBS volume that had all the Maven central repository within it. Does anyone know of such a thing? thanks, Mark - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org