On Wed, Jul 25, 2012 at 4:19 PM, Brett Porter <br...@apache.org> wrote:
> On 26/07/2012, at 3:46 AM, Brian Fox wrote: > > > On Tue, Jul 24, 2012 at 7:16 PM, Brett Porter <br...@apache.org> wrote: > > > >> > >> My understanding is that unfortunately Sonatype are not allowing anyone > >> else to mirror the content directly any more. > >> > > > > Ibiblio disabled the rsync on their own accord because it was thrashing > > their disks. > > That doesn't seem consistent with Joel's response on the ticket. When did > that happen? > > If this is the case, we should establish another "escrow" location. > > > Let me clarify my statement because it's not opposed to what Joel said, we're talking about different things. Ibiblio turned off OUTBOUND rsyncs because too many people where thrashing their disks and chewing up massive amounts of bandwidth. The INBOUND rsync is still happenning. Because Central is over 600GB now, even the rsync from us was thrashing both of our disks so Joel developed a new method that works like this: Central Staging reaches out to all the forges and does mini rsyncs scoped by each groupId folder so they aren't causing any real load. We take the log output from those to get a manifest of the files that changed. They get added all up and then inserted into the include-files portion of the outbound rsync to the other central machines and to ibiblio. The net effect is that it reduces the churn that rsync produces on the systems, speeds up the sync and still keeps every file updated. Since the outbound sync doesn't touch that last_updated.txt file, it isn't being added as a changed file to any of the rsync pushes, hence it appears outdated on Ibiblio. But if you poke around out there, you'll find that the components are in fact being updated. With the move to the CDN, we have moved a massive amount of traffic back off of Ibiblio because they were previously serving the indexes. I'll ping them and see if the rsync is possible again.
