Re: [CentOS] Local mirroring of the CentOS repos
Kai Schaetzl wrote: Ashley M. Kirchner wrote on Fri, 12 Oct 2007 08:59:53 -0600: Uhhh, I don't experience that when I run my rsync script. It actually does delete stuff that's no longer on the mirror. Yeah, but the repo seems to keep a lot ... I downloaded 5 or more versions of the same rpm of some software (for instance all the kernel packages, openoffice stuff in three versions) because there were two months or so between my first retrieval and the second (as I didn't have time to continue earlier) So, somewhere, someone is cleaning out the mirrors of old stuff. Now, an argument for how often this happens might be more like it, but it does happen. Ah, well, so the CentOS 5 updates repo hasn't been cleaned yet? In that case they didn't clean it since starting it. From looking at the CentOS 4 repo's it looks like they don't clean, though. Repos get automatically cleaned when moving over from x.1 to x.2 etc. So, when 5.1 arrives the repo will have been cleared by then and start anew. Don't get me wrong, I'm not complaining that they keep all those files. There may be good reasons to do so, I don't know. I merely wanted to see if I can do something from my side to avoid getting it all. Apparently, I can avoid it when regularly downloading by specifying a timelimit, but I can't avoid getting it all when I start a mirror (unless I use a filelist). Kai Why don't you want to keep old updates? What if you need to roll back because of some bad bug introduced in the latest version? The you have to go back to get either the original or the update that came before the version you have. And also, don't say mirror if you don't want to have the same packages. Just say you want to create a new repository containing some of the packages from the centos mirror. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
On Thu, 11 Oct 2007, Lorenzo Quatrini wrote: Kai Schaetzl ha scritto: Lorenzo wrote on Thu, 11 Oct 2007 09:38:21 +0200: Have you tried mrepo? How would this help? The main problem is to get rid of the old updates. Kai You're right, I thought that mrepo would get rid of old updates by himself, but id doesn't. I am looking right now on different ways to get the same goal (save bandwidth, time and disk space); if I find someting I'll post on the mailing list. The problem is not getting rid of the old updates. I can write the functionality to do that. The problem is the fact that when you mirror something, you get a complete copy of the mirror. The copy-tool (whether it is lftp, rsync or something else) has no notion of 'only the recent package'. It has no notion of versions or packages. So while we could remove the older updates, you would pull them in every time again and remove them again. I am certain this is what nobody wants :) However, if you use RHN with the rhnget tool, it does have the notion of downloading only the recent updates. So the synchronisation with RHN allows you to specify to clean up the old updates because the tool works on RPMs. Not just on files. If someone can come up with a smart way of handling this, mrepo can do it. -- -- dag wieers, [EMAIL PROTECTED], http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Kai Schaetzl wrote: Apparently, I can avoid it when regularly downloading by specifying a timelimit, but I can't avoid getting it all when I start a mirror (unless I use a filelist). That wouldn't be a mirror then, would it? :) I suppose, if you're really wanting to, one thing you could attempt, is what you suggested and maintain a filelist of what you want (or don't want.) Another possibility might be to roll your own script to run rsync in a dry-run mode (-n) and filter the output, then go back and grab what you want based on that. The dry-run mode will simply tell you what rsync would transfer without actually doing it. So you can then take the result, parse it, figureout by version numbers what you want, and then fetch it all, again through rsync. But honestly, at that point, I just transfer the whole thing and not worry about it. It's not like it takes an extremely large amount of space, at least not to me: :~ du -h --max-depth=2 CentOS/ 110MCentOS/5.0/extras 3.4GCentOS/5.0/updates 32K CentOS/5.0/addons 3.5GCentOS/5.0 3.5GCentOS/ (I exclude everything else from my rsync transfer, 64bit, isos, etc., etc.) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Ashley M. Kirchner wrote on Fri, 12 Oct 2007 08:59:53 -0600: Uhhh, I don't experience that when I run my rsync script. It actually does delete stuff that's no longer on the mirror. Yeah, but the repo seems to keep a lot ... I downloaded 5 or more versions of the same rpm of some software (for instance all the kernel packages, openoffice stuff in three versions) because there were two months or so between my first retrieval and the second (as I didn't have time to continue earlier) So, somewhere, someone is cleaning out the mirrors of old stuff. Now, an argument for how often this happens might be more like it, but it does happen. Ah, well, so the CentOS 5 updates repo hasn't been cleaned yet? In that case they didn't clean it since starting it. From looking at the CentOS 4 repo's it looks like they don't clean, though. Repos get automatically cleaned when moving over from x.1 to x.2 etc. So, when 5.1 arrives the repo will have been cleared by then and start anew. Don't get me wrong, I'm not complaining that they keep all those files. There may be good reasons to do so, I don't know. I merely wanted to see if I can do something from my side to avoid getting it all. Apparently, I can avoid it when regularly downloading by specifying a timelimit, but I can't avoid getting it all when I start a mirror (unless I use a filelist). Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Kai Schaetzl ha scritto: I figured I try if I can mirror the base and updates repos locally. There's no tutorial for that, only one about creating your own repo of packages which is not the same. So, I just mirrored all the stuff with wget and changed the baseurl in the repo files and hoped that's enough. Works. So easy you don't need a tutorial. *But* I then realized that the updates directory contains *all* updates, not just the latest. Which means if I don't regularly check I may get old versions mirrored I don't want. It also means that I get a lot of unwanted files at the time I start to mirror. And I cannot delete old files as these would again be mirrored in. An obvious solution would be to check each day and tell wget (or whatever software I use) to ignore files older than 24 hours. Still, this means the initial download has to get them all and I have to delete all unwanted old files manually. Is there a better solution? Kai Have you tried mrepo? Regards Lorenzo Quatrini ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Lorenzo wrote on Thu, 11 Oct 2007 09:38:21 +0200: Have you tried mrepo? How would this help? The main problem is to get rid of the old updates. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Mark pryor wrote on Wed, 10 Oct 2007 13:30:46 -0700 (PDT): I only have a comment about the base mirror. Instead of using the internet to make a base mirror (not sure you did it that way), you can use the CentOS-Media.repo Ah, well, I remember having read about this, but admit I didn't think about it. In my case it wouldn't be usable, though. I want to setup several VMs and several physical machines and thought it would be stupid to update them all from a remote mirror. I set up the base repo by just copying the relevant stuff from the DVD to the harddisk of my Win2k3 server and then retrieved updates via wget from a mirror in Germany. Now I can install the VMs from my local mirror via FTP (installing from http won't work, don't know why) and update them via yum (using HTTP, it seems FTP doesn't work with yum) from the local mirror. The physical machines get a DVD drive attached, get a minimal install, DVD drive removed and then I can also update and add from the local mirror. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
On Thu, 11 Oct 2007, Lorenzo Quatrini wrote: Kai Schaetzl ha scritto: Lorenzo wrote on Thu, 11 Oct 2007 09:38:21 +0200: Have you tried mrepo? How would this help? The main problem is to get rid of the old updates. Kai Back in the old days, I used to use autoupdate .. and I believe that it would update the rpms you had and not keep the old ones. Autoupdate is available at http://www.mat.univie.ac.at/~gerald/ftp/autoupdate/. Check this out and let us know if this works for you. Barry ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Kai Schaetzl ha scritto: Lorenzo wrote on Thu, 11 Oct 2007 09:38:21 +0200: Have you tried mrepo? How would this help? The main problem is to get rid of the old updates. Kai You're right, I thought that mrepo would get rid of old updates by himself, but id doesn't. I am looking right now on different ways to get the same goal (save bandwidth, time and disk space); if I find someting I'll post on the mailing list. Regards Lorenzo ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Kai Schaetzl wrote: Thanks, but this will still download and keep *all* updates for a platform. If they exist on the mirror, then yes. However, anything that gets removed from the mirror you're using, will also get deleted from your local copy. That's the whole idea behind rsync's --delete flag. In my case, I don't really care if the CentOS mirrors don't delete old stuff, I just pull everything down. However, on the Fedora side, they do remove old packages, and rsync will automatically remove them from the local copy as well. I backup some 15 servers that way. They're all going to a single backup server where I keep up to 6 weeks worth of data (for some, 2 weeks on others) using rsync as a backup utility. Using hardlinks between backups allows me to keep that much information. And if a file gets removed from the source, it will get removed from the backup as well, but only for that run, not the previous ones. As for not having rsync on Win2K, you can install CygWin on it and use rsync then. I have a Server2003 pulling 2 TiB of data every night from an old Win2K server using rsync. -- W | It's not a bug - it's an undocumented feature. + Ashley M. Kirchner mailto:[EMAIL PROTECTED] . 303.442.6410 x130 IT Director / SysAdmin / Websmith . 800.441.3873 x130 Photo Craft Imaging . 3550 Arapahoe Ave. #6 http://www.pcraft.com . . .. Boulder, CO 80303, U.S.A. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
RE: [CentOS] Local mirroring of the CentOS repos
Kai Schaetzl wrote: I figured I try if I can mirror the base and updates repos locally. There's no tutorial for that, only one about creating your own repo of packages which is not the same. So, I just mirrored all the stuff with wget and changed the baseurl in the repo files and hoped that's enough. Works. So easy you don't need a tutorial. *But* I then realized that the updates directory contains *all* updates, not just the latest. Which means if I don't regularly check I may get old versions mirrored I don't want. It also means that I get a lot of unwanted files at the time I start to mirror. And I cannot delete old files as these would again be mirrored in. An obvious solution would be to check each day and tell wget (or whatever software I use) to ignore files older than 24 hours. Still, this means the initial download has to get them all and I have to delete all unwanted old files manually. Is there a better solution? Yes a lot of past versions are kept in the repo, but if you filter those out then it wouldn't be a mirror then? Be careful with the repomd.xml file, delete it before starting to mirror to make sure it doesn't get out-of-sync, because, at least on my end, I find that wget via http doesn't compare time stamps just sizes/names and the sha1 checksums are always the same length. Here's a little script: #!/bin/sh VERSION=5.0 mkdir -p /Software/CentOS/$VERSION/updates/i386 /dev/null 21 mkdir -p /Software/CentOS/$VERSION/updates/x86_64 /dev/null 21 rm -f /Software/CentOS/$VERSION/updates/i386/repodata/repomd.xml /dev/null 21 rm -f /Software/CentOS/$VERSION/updates/x86_64/repodata/repomd.xml /dev/null 21 wget -nH --cache=off --cut-dirs=4 -m -c -R gif,png,^index.html* -I ^/centos/$VERSION/updates/i386/ -P /Software/CentOS/$VERSION/updates/i386 http://mirror.centos.org/centos/$VERSION/updates/i386/ wget -nH --cache=off --cut-dirs=4 -m -c -R gif,png,^index.html* -I ^/centos/$VERSION/updates/x86_64/ -P /Software/CentOS/$VERSION/updates/x86_64 http://mirror.centos.org/centos/$VERSION/updates/x86_64/ -Ross __ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Kai Schaetzl [EMAIL PROTECTED] wrote: I figured I try if I can mirror the base and updates repos locally. There's no tutorial for that, only one about creating your own repo of packages which is not the same. So, I just mirrored all the stuff with wget and changed the baseurl in the repo files and hoped that's enough. Works. So easy you don't need a tutorial. Kai, I only have a comment about the base mirror. Instead of using the internet to make a base mirror (not sure you did it that way), you can use the CentOS-Media.repo This works best if you have the DVD ISO #mkdir /mnt/C564 #nano /etc/fstab --- add at end /path-to/CentOS-5.0-x86_64-bin-DVD.iso /mnt/C564 iso9660 ro,loop,async 0 0 --- unsnip - now edit /etc/yum.repos.d/CentOS-Media.repo and add to [c5-media] file:///mnt/C564 then you want YUM to ignore the [base] repo and use [c5-media] #yum search some-rpm --disablerepo=base --enablerepo=c5-media now you have speed and still have all the default abilities of YUM to find the RPM owner of [c5-media]: [EMAIL PROTECTED] ~]$ rpm -qf /etc/yum.repos.d/CentOS-Media.repo centos-release-5-0.0.el5.centos.2 My repo for C5 (mpryor-c5.repo) at http://www.tlviewer.org/centos -- Mark - Check out the hottest 2008 models today at Yahoo! Autos.___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Local mirroring of the CentOS repos
Kai Schaetzl wrote: *But* I then realized that the updates directory contains *all* updates, not just the latest. Which means if I don't regularly check I may get old versions mirrored I don't want. It also means that I get a lot of unwanted files at the time I start to mirror. And I cannot delete old files as these would again be mirrored in. An obvious solution would be to check each day and tell wget (or whatever software I use) to ignore files older than 24 hours. Still, this means the initial download has to get them all and I have to delete all unwanted old files manually. Is there a better solution? Use rsync. I keep a local copy of the updates, specific to my platform. I'm also very specific in WHAT I want locally as you'll see in the following script. I use the Stanford University's mirror. :~ cat rsync.sh echo Getting Centos 5.0 Updates...; echo -e *\n; rsync --progress --archive \ --partial --delete --delete-excluded \ --exclude centosplus/ \ --exclude fasttrack/ \ --exclude isos/ \ --exclude isos-dvd/ \ --exclude os/ \ --exclude updates/SRPMS/ \ --exclude updates/x86_64/ \ --exclude addons/SRPMS/ \ --exclude addons/x86_64/ \ --exclude extras/SRPMS/ \ --exclude extras/x86_64/ \ mirror.stanford.edu::mirrors/centos/5.0 /home/CentOS/; -- W | It's not a bug - it's an undocumented feature. + Ashley M. Kirchner mailto:[EMAIL PROTECTED] . 303.442.6410 x130 IT Director / SysAdmin / Websmith . 800.441.3873 x130 Photo Craft Imaging . 3550 Arapahoe Ave. #6 http://www.pcraft.com . . .. Boulder, CO 80303, U.S.A. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
