Re: mirroring question
On Fri, Aug 09, 2002 at 04:03:05PM -0400, Loren Jordan wrote: The question I have is, how can I get a mirror copy over http only? There are a couple of apt-get'able archives I use parts of and would like to mirror them as well for my own use. The problem is, they only allow http access. I have tried wget in mirror mode but I have yet to be able to keep it from wandering outside of the specific section I want to mirror. It also gets all the possible index versions (Apache index sorted by each possible thing) and just makes thing rather cluttered. [As if this thread didn't already have enough shameless plugs] I've got a strange partial mirror tool I wrote which uses libapt. It can use any method supported by libapt (http, ftp, copy, ...) and uses a sources.list file to specify what archives to mirror. Since it was originally intended just to let me 'fetch' packages without pulling down dependencies, I named it apt-fetch. apt-fetch fetch [package list] - same as apt-get install -d except it does no dependency checking. apt-fetch sync - syncs up the mirror. Uses a dpkg 'status' file to determine which packages should be held (not downloaded). This could probably be mixed with one of those apt-proxy type tools to select packages to mirror based on which packages have been requested. There's a simple shell script which does things like set APT_CONFIG, clean out dead links and call apt-fetch with various options. apt-fetch itself is written in c++ and isn't exactly the prettiest batch of code I've ever written. The tool has been used to keep a campus partial mirror going for close to two years now. It has some limitations (only one architecture at a time, doesn't grab the Release files, etc), but is otherwise pretty nice. It downloads everything into a single directory and then makes hard links to create the actual mirror structure, which may be a little inelegant, but it works and allows me to clean out old packages with apt-get autoclean. If you just want to maintain a small mirror and generate your own Packages files, then it's actually a good solution. I haven't touched the code in over a year, but if anyone thinks this might be a useful tool and wants to test it or take it over, let me know. - Ray (duster on #debian)
Re: mirroring question [almost solved]
At 10:09 PM 8/9/2002 +0200, Mathieu.Segaud wrote: There is a patch for a debmirror for mirroring over http Attached to this mail. Regards, Mathieu Segaud This patch solved my problem! I was hoping that I wasn't going to re-write my script for this and I didn't have to. Thanks a lot for all the responses. Now the next question regarding debmirror... It appears that it will only retrieve the contents of a directory as they are listed in the Packages.gz or Sources.gz files. With the default --cleanup setting, it destroyed the woody/main/disk-i386 directory from my mirror. I only enabled that setting after deciding that everything was working (I didn't want to re-download the entire archive...) I only lost the disks-i386 directory, and now have fixed things with a fmirror of that directory and a little interesting linking. My mirror now looks like it should except I have to re-fix the links to disks-i386 every time I sync my mirror to the real one. This is automated but should not be needed. Is there any plans to add a feature to debmirror to grab entire sub directories so it will be possible to keep things like disks-i386 and even /debian/tools so rawrite.exe is in the mirror. Thanks again! Loren
Re: mirroring question
On Fri, Aug 09, 2002 at 04:03:05PM -0400, Loren Jordan wrote: I just want to pull a copy of these packages just like they sit in the apt-get'able directory structure so I don't have to re-create the Packages[.gz] files and such... use apt-move, apt-proxy or demish (http://luon.net/~admar) apt-move and demish are really mirror programs that rely on apt for downloading of packages, while apt-proxy is 'just' a cache (you could also try squid instead of apt-proxy). /shamelessly promoting his own package demish builds a mirror, containing only those packages (and optionally their dependencies) that the maintainer wants, and thus gives him (almost) full control over the size of the mirror. It is useful if you have a fast internet connection at work/school and a laptop or a cdwriter: just make a mirror on the laptop or cd, and update your computers at home. /end of commercial block Cheers! Admar
Re: mirroring question
apt-move and demish are really mirror programs that rely on apt for downloading of packages, while apt-proxy is 'just' a cache (you could also try squid instead of apt-proxy). another shameless plug Or you could try Apt-cacher: http://www.apt-cacher.com/ /another shameless plug Jonathan Oxer Ph +61 3 9723 9399 / Fx +61 3 9723 4899 GPG key: http://www.ivt.com.au/gpg/jon.oxer.gpg
Re: mirroring question
Oops! another shameless plug Or you could try Apt-cacher: http://www.apt-cacher.com/ /another shameless plug http://www.apt-cacher.org/
