On Tue, Nov 23, 2010 at 10:24:18PM +0100, David Landgren wrote: > On 22/11/2010 15:18, David Nicol wrote: > >On Mon, Nov 22, 2010 at 4:37 AM, David Landgren<da...@landgren.net> wrote: > >>Yeah, this is the killer. In an ideal world, we would kill the symlinks > >>such > >>as authors/id/*, modules/by-category/*, modules/by-module/* and so on. > >>These > >>could be recreated via shell scripts locally on mirrors for people who > >>wish > >>to maintain these legacies. Cutting that out would diminish the rsync > >>burden > >>considerably. > >> > >>David > > > >or re-engineer CPAN as a sqlite+FTSE database, and re-engineer the > >mirroring process as a database mirror via a TBD compact database diff > >protocol (I have no intention of doing any of this myself; good > >morning) > > Well... I guess that's not going to happen then, is it? > > I shouldn't even bother replying, but I wouldn't want the archives to > think that silence indicates tacit agreement.
The symlink tree is built by scripts, isn't it? Are they available? Because the nice thing about your suggestion is that it doesn't involve changing any of the server infrastructure, and it's an incremental change which can be done by each mirror in turn. Instead of running rsync over the whole tree, it can change to run a top level script that runs rsync over the parts that have to be copied, and then run the symlink generation on the parts that can be recreated locally. Nicholas Clark
