Re: Maven2 mirror @ netcologne
On Wed, Jul 25, 2012 at 4:19 PM, Brett Porter wrote: > On 26/07/2012, at 3:46 AM, Brian Fox wrote: > > > On Tue, Jul 24, 2012 at 7:16 PM, Brett Porter wrote: > > > >> > >> My understanding is that unfortunately Sonatype are not allowing anyone > >> else to mirror the content directly any more. > >> > > > > Ibiblio disabled the rsync on their own accord because it was thrashing > > their disks. > > That doesn't seem consistent with Joel's response on the ticket. When did > that happen? > > If this is the case, we should establish another "escrow" location. > > > Let me clarify my statement because it's not opposed to what Joel said, we're talking about different things. Ibiblio turned off OUTBOUND rsyncs because too many people where thrashing their disks and chewing up massive amounts of bandwidth. The INBOUND rsync is still happenning. Because Central is over 600GB now, even the rsync from us was thrashing both of our disks so Joel developed a new method that works like this: Central Staging reaches out to all the forges and does mini rsyncs scoped by each groupId folder so they aren't causing any real load. We take the log output from those to get a manifest of the files that changed. They get added all up and then inserted into the include-files portion of the outbound rsync to the other central machines and to ibiblio. The net effect is that it reduces the churn that rsync produces on the systems, speeds up the sync and still keeps every file updated. Since the outbound sync doesn't touch that last_updated.txt file, it isn't being added as a changed file to any of the rsync pushes, hence it appears outdated on Ibiblio. But if you poke around out there, you'll find that the components are in fact being updated. With the move to the CDN, we have moved a massive amount of traffic back off of Ibiblio because they were previously serving the indexes. I'll ping them and see if the rsync is possible again.
Re: Maven2 mirror @ netcologne
On 26/07/2012, at 3:46 AM, Brian Fox wrote: > On Tue, Jul 24, 2012 at 7:16 PM, Brett Porter wrote: > >> >> My understanding is that unfortunately Sonatype are not allowing anyone >> else to mirror the content directly any more. >> > > Ibiblio disabled the rsync on their own accord because it was thrashing > their disks. That doesn't seem consistent with Joel's response on the ticket. When did that happen? If this is the case, we should establish another "escrow" location. - Brett -- Brett Porter br...@apache.org http://brettporter.wordpress.com/ http://au.linkedin.com/in/brettporter http://twitter.com/brettporter - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Maven2 mirror @ netcologne
Hello, > Central is now on a CDN so parallel mirrors don't really help anymore. See > more here > http://www.sonatype.com/people/2012/07/we-just-kicked-central-performance-and-availability-up-a-notch-with-edgecast/ Is there a way to retrieve a full copy or to mirror Maven Central from your new CDN (with rsync, or others)? I would like to build a service that uses these artifacts but it seems really complex or even impossible to retrieve update to date information about artifacts. I mean that every two hours new artifacts may be published but it seems that nexus indexes are updated only every week. Also, the maven search engine [1] seems to be synchronized with an update frequency lower than 4 hours. Is it not possible to offer to everyone the same possibility (more frequent update of indexes or update through rsync, etc.)? [1] http://search.maven.org/ Kind Regards, Laurent On Wed, Jul 25, 2012 at 7:46 PM, Brian Fox wrote: > On Tue, Jul 24, 2012 at 7:16 PM, Brett Porter wrote: > >> >> My understanding is that unfortunately Sonatype are not allowing anyone >> else to mirror the content directly any more. >> > > Ibiblio disabled the rsync on their own accord because it was thrashing > their disks. > > Central is now on a CDN so parallel mirrors don't really help anymore. See > more here > http://www.sonatype.com/people/2012/07/we-just-kicked-central-performance-and-availability-up-a-notch-with-edgecast/ - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Maven2 mirror @ netcologne
On Tue, Jul 24, 2012 at 7:16 PM, Brett Porter wrote: > > My understanding is that unfortunately Sonatype are not allowing anyone > else to mirror the content directly any more. > Ibiblio disabled the rsync on their own accord because it was thrashing their disks. Central is now on a CDN so parallel mirrors don't really help anymore. See more here http://www.sonatype.com/people/2012/07/we-just-kicked-central-performance-and-availability-up-a-notch-with-edgecast/
Re: Maven2 mirror @ netcologne
On 24/07/2012, at 9:39 PM, Christian Rohmann wrote: > Hello again, > > 1) I noticed that the rsync for the maven2 repository > "mirrors.ibiblio.org::maven2" does not work anymore. This is supposed to work, so I opened an issue: https://issues.sonatype.org/browse/MVNCENTRAL-225. I cloned the previous instance where it was broken for a significant amount of time. > Anyways, mirror.netcologne.de has now some powerful new hardware and > lots of space. We would love to get the maven2 mirror working again and > serve as a mirror for users in Germany and Europe to be able to fetch > files faster. See the attached email with the inital offer to mirror the > maven2 repo. We now have a way more powerful box with more than 20 TB of > space, 48GB of RAM and 2 gbit/s of IPv4 and IPv6 connectity. My understanding is that unfortunately Sonatype are not allowing anyone else to mirror the content directly any more. > 2) I desperately try to sign up to repo-maintain...@maven.apache.org to > ask there, but after I send back the "confirm subscribe" I don't get an > answer from the mailing list. Are you manually accepting subscriptions? Yes, it had moderated subscription - we're overdue to shut that list down since there's nothing the Maven team can do to answer these questions any more. Did you see an outstanding reference to it on the web site? - Brett -- Brett Porter br...@apache.org http://brettporter.wordpress.com/ http://au.linkedin.com/in/brettporter http://twitter.com/brettporter - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Maven2 mirror @ netcologne
I've updated the metadata: http://repo1.maven.org/maven2/.meta/repository-metadata.xml On Mon, Jan 4, 2010 at 2:08 PM, Carlos Sanchez wrote: > Hi Christian, > > Thanks for providing the repo! > Please subscribe to repo-maintain...@maven.apache.org, that's the list > for repository matters. > > The mirrors are updated once a day, so you can reduce the frequency of > your sync. > > Cheers > > > On Fri, Jan 1, 2010 at 2:51 AM, Christian Rohmann > wrote: >> Hello again, >> >> I wrote you in march (see attached email) aber setting up a maven2 mirror in >> Germany. Unfortunately I never heard back from you. >> >> The official repo is rather slow when accessed from Germany (due to small >> many files and latency I guess), maybe we could help out with our mirror >> here. >> >> If you are interested in the offer, please feel free to list us with the >> other official mirrors. >> >> >> >> Happy New Year and Regards from Germany, >> >> -- >> Christian Rohmann >> >> Content Delivery Server u. Dienste >> Network Engineering & Design >> >> >> NETCOLOGNE Gesellschaft für Telekommunikation mbH >> Am Coloneum 9 | 50829 Köln >> Tel: 0221 -5751 | Fax: 0221 -75751 >> http://www.netcologne.de >> >> Geschäftsführer: >> Werner Hanf >> Dipl.-Ing. Karl-Heinz Zankel >> HRB 25580, AG Köln >> >> >> -- Forwarded message -- >> From: Mirror-Service >> To: ...@maven.apache.org >> Date: Thu, 05 Mar 2009 09:10:56 +0100 >> Subject: Maven2 mirror @ netcologne >> Hello Maven Dev-Team, >> >> we would love to setup a mirror of the maven2 repository in Germany. >> As there are currently no mirrors in Germany at all, and only two danish >> mirrors in europe. >> >> The NetCologne GmbH is a local ISP in the greater Cologne area. Our >> machine "mirror.netcologne.de" currently runs on this hardware: >> >> 2x 3Ghz Intel Xeon CPU >> 2 GB RAM (to be upgraded to 4GB) >> 3 TB of mirror storage total >> >> The machine currently already mirrors a few linux distros (mainly >> debian) and other free software projects. >> >> Regarding internet connectivity, the machine itself has 2 Gigabit/s >> uplink. Our backbone is then connected to all major peering points in >> europe (DECIX, AMSIX, LINX) with 20 Gbit/s each. We are also peering in >> the US (2x EQUINIX) and maintain quite a few private peering with other >> ISPs including Deutsche Telekom (DTAG) and the DFN (Deutsches >> Forschungsnetz, the German university network). >> >> IPv6 connectivity is coming very soon, so mark that as checked. >> >> We configured the rsync to sync with: "mirrors.ibiblio.org::maven2" >> every two hours in the 42nd minute. Please let me know if that's ok, or >> if we need to adjust that. The repo is already available on >> [rsync|ftp|http]://mirror.netcologne.de/maven2 >> >> >> If you need to contact me, just mail to crohm...@netcologne.de, as >> "official" contact for the mirror use "mirror-serv...@netcologne.de". >> >> >> >> -- >> Mit freundlichen Grüßen >> >> Christian Rohmann >> >> --- >> Network Engineering and Design >> >> Tel.: +49 221 -5751 >> Fax.: +49 221 -75751 >> >> NetCologne Gesellschaft für Telekommunikation mbH >> Am Coloneum 9 >> 50829 Köln >> >> Geschäftsführer: Werner Hanf, Karl- Heinz Zankel, Dipl. Ing. >> HRB 25580, AG Köln >> www.netcologne.de >> >> >> >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> For additional commands, e-mail: dev-h...@maven.apache.org >> > - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Maven2 mirror @ netcologne
Hi Christian, Thanks for providing the repo! Please subscribe to repo-maintain...@maven.apache.org, that's the list for repository matters. The mirrors are updated once a day, so you can reduce the frequency of your sync. Cheers On Fri, Jan 1, 2010 at 2:51 AM, Christian Rohmann wrote: > Hello again, > > I wrote you in march (see attached email) aber setting up a maven2 mirror in > Germany. Unfortunately I never heard back from you. > > The official repo is rather slow when accessed from Germany (due to small > many files and latency I guess), maybe we could help out with our mirror > here. > > If you are interested in the offer, please feel free to list us with the > other official mirrors. > > > > Happy New Year and Regards from Germany, > > -- > Christian Rohmann > > Content Delivery Server u. Dienste > Network Engineering & Design > > > NETCOLOGNE Gesellschaft für Telekommunikation mbH > Am Coloneum 9 | 50829 Köln > Tel: 0221 -5751 | Fax: 0221 -75751 > http://www.netcologne.de > > Geschäftsführer: > Werner Hanf > Dipl.-Ing. Karl-Heinz Zankel > HRB 25580, AG Köln > > > -- Forwarded message -- > From: Mirror-Service > To: ...@maven.apache.org > Date: Thu, 05 Mar 2009 09:10:56 +0100 > Subject: Maven2 mirror @ netcologne > Hello Maven Dev-Team, > > we would love to setup a mirror of the maven2 repository in Germany. > As there are currently no mirrors in Germany at all, and only two danish > mirrors in europe. > > The NetCologne GmbH is a local ISP in the greater Cologne area. Our > machine "mirror.netcologne.de" currently runs on this hardware: > > 2x 3Ghz Intel Xeon CPU > 2 GB RAM (to be upgraded to 4GB) > 3 TB of mirror storage total > > The machine currently already mirrors a few linux distros (mainly > debian) and other free software projects. > > Regarding internet connectivity, the machine itself has 2 Gigabit/s > uplink. Our backbone is then connected to all major peering points in > europe (DECIX, AMSIX, LINX) with 20 Gbit/s each. We are also peering in > the US (2x EQUINIX) and maintain quite a few private peering with other > ISPs including Deutsche Telekom (DTAG) and the DFN (Deutsches > Forschungsnetz, the German university network). > > IPv6 connectivity is coming very soon, so mark that as checked. > > We configured the rsync to sync with: "mirrors.ibiblio.org::maven2" > every two hours in the 42nd minute. Please let me know if that's ok, or > if we need to adjust that. The repo is already available on > [rsync|ftp|http]://mirror.netcologne.de/maven2 > > > If you need to contact me, just mail to crohm...@netcologne.de, as > "official" contact for the mirror use "mirror-serv...@netcologne.de". > > > > -- > Mit freundlichen Grüßen > > Christian Rohmann > > --- > Network Engineering and Design > > Tel.: +49 221 -5751 > Fax.: +49 221 -75751 > > NetCologne Gesellschaft für Telekommunikation mbH > Am Coloneum 9 > 50829 Köln > > Geschäftsführer: Werner Hanf, Karl- Heinz Zankel, Dipl. Ing. > HRB 25580, AG Köln > www.netcologne.de > > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
RE: Maven2 mirror @ netcologne
Ibiblio only gets updated once a day starting at 1am CST, so every 2 hours is overkill. You could wait until 2am CST and then pull your copy. -Original Message- From: Mirror-Service [mailto:mirror-serv...@netcologne.de] Sent: Thursday, March 05, 2009 3:11 AM To: dev@maven.apache.org Subject: Maven2 mirror @ netcologne Hello Maven Dev-Team, we would love to setup a mirror of the maven2 repository in Germany. As there are currently no mirrors in Germany at all, and only two danish mirrors in europe. The NetCologne GmbH is a local ISP in the greater Cologne area. Our machine "mirror.netcologne.de" currently runs on this hardware: 2x 3Ghz Intel Xeon CPU 2 GB RAM (to be upgraded to 4GB) 3 TB of mirror storage total The machine currently already mirrors a few linux distros (mainly debian) and other free software projects. Regarding internet connectivity, the machine itself has 2 Gigabit/s uplink. Our backbone is then connected to all major peering points in europe (DECIX, AMSIX, LINX) with 20 Gbit/s each. We are also peering in the US (2x EQUINIX) and maintain quite a few private peering with other ISPs including Deutsche Telekom (DTAG) and the DFN (Deutsches Forschungsnetz, the German university network). IPv6 connectivity is coming very soon, so mark that as checked. We configured the rsync to sync with: "mirrors.ibiblio.org::maven2" every two hours in the 42nd minute. Please let me know if that's ok, or if we need to adjust that. The repo is already available on [rsync|ftp|http]://mirror.netcologne.de/maven2 If you need to contact me, just mail to crohm...@netcologne.de, as "official" contact for the mirror use "mirror-serv...@netcologne.de". -- Mit freundlichen Grüßen Christian Rohmann --- Network Engineering and Design Tel.: +49 221 -5751 Fax.: +49 221 -75751 NetCologne Gesellschaft für Telekommunikation mbH Am Coloneum 9 50829 Köln Geschäftsführer: Werner Hanf, Karl- Heinz Zankel, Dipl. Ing. HRB 25580, AG Köln www.netcologne.de - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
