On Fri, Aug 09, 2002 at 04:03:05PM -0400, Loren Jordan wrote: > The question I have is, how can I get a mirror copy over http only? There > are a couple of apt-get'able archives I use parts of and would like to > mirror them as well for my own use. The problem is, they only allow http > access. I have tried wget in "mirror" mode but I have yet to be able to > keep it from wandering outside of the specific section I want to > mirror. It also gets all the possible index versions (Apache index sorted > by each possible thing) and just makes thing rather cluttered.
[As if this thread didn't already have enough shameless plugs] I've got a strange partial mirror tool I wrote which uses libapt. It can use any method supported by libapt (http, ftp, copy, ...) and uses a sources.list file to specify what archives to mirror. Since it was originally intended just to let me 'fetch' packages without pulling down dependencies, I named it apt-fetch. apt-fetch fetch [package list] - same as "apt-get install -d" except it does no dependency checking. apt-fetch sync - syncs up the mirror. Uses a dpkg 'status' file to determine which packages should be "held" (not downloaded). This could probably be mixed with one of those apt-proxy type tools to select packages to mirror based on which packages have been requested. There's a simple shell script which does things like set APT_CONFIG, clean out dead links and call apt-fetch with various options. apt-fetch itself is written in c++ and isn't exactly the prettiest batch of code I've ever written. The tool has been used to keep a campus partial mirror going for close to two years now. It has some limitations (only one architecture at a time, doesn't grab the Release files, etc), but is otherwise pretty nice. It downloads everything into a single directory and then makes hard links to create the actual mirror structure, which may be a little inelegant, but it works and allows me to clean out old packages with "apt-get autoclean". If you just want to maintain a small mirror and generate your own Packages files, then it's actually a good solution. I haven't touched the code in over a year, but if anyone thinks this might be a useful tool and wants to test it or take it over, let me know. - Ray (duster on #debian)