neysx 06/05/12 20:07:07 Modified: rsync.xml Log: Fixed utterly broken coding style and a couple of typos
Revision Changes Path 1.49 xml/htdocs/doc/en/rsync.xml file : http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/rsync.xml?rev=1.49&content-type=text/x-cvsweb-markup&cvsroot=gentoo plain: http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/rsync.xml?rev=1.49&content-type=text/plain&cvsroot=gentoo diff : http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/rsync.xml.diff?r1=1.48&r2=1.49&cvsroot=gentoo Index: rsync.xml =================================================================== RCS file: /var/cvsroot/gentoo/xml/htdocs/doc/en/rsync.xml,v retrieving revision 1.48 retrieving revision 1.49 diff -u -r1.48 -r1.49 --- rsync.xml 12 May 2006 12:59:39 -0000 1.48 +++ rsync.xml 12 May 2006 20:07:07 -0000 1.49 @@ -1,25 +1,19 @@ <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE guide SYSTEM "/dtd/guide.dtd"> -<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/rsync.xml,v 1.48 2006/05/12 12:59:39 jforman Exp $ --> +<!-- $Header: /var/cvsroot/gentoo/xml/htdocs/doc/en/rsync.xml,v 1.49 2006/05/12 20:07:07 neysx Exp $ --> <guide link="/doc/en/rsync.xml" lang="en"> <title>Gentoo Linux rsync Mirrors Policy and Guide</title> -<author title="Author, new version"> - Tobias Klausmann -</author> - -<author title="Original Author"> +<author title="Author"> <mail link="[EMAIL PROTECTED]">Gentoo Mirror Administrators</mail> </author> -<author title="Original Editor"> - <mail link="[EMAIL PROTECTED]">Xavier Neys</mail> -</author> -<author title="Contributor"> +<author title="Author"> <mail link="[EMAIL PROTECTED]">Tobias Klausmann</mail> </author> - - +<author title="Editor"> + <mail link="[EMAIL PROTECTED]">Xavier Neys</mail> +</author> <abstract> This document explains how to set up a official rsync mirror and your own local @@ -30,7 +24,7 @@ <!-- See http://creativecommons.org/licenses/by-sa/2.5 --> <license/> -<version>3.0</version> +<version>3.1</version> <date>2006-05-12</date> <chapter> @@ -41,21 +35,24 @@ <p> This guide is intended for people who would like to set up an rsync mirror of -their own. I caters not only to those who want to run an official rsync mirror +their own. It caters not only to those who want to run an official rsync mirror but also those wanting to run private mirrors. </p> -<p>There are three kinds of Gentoo rsync mirrors: main rotation mirrors, -community mirrors and private mirrors. Main rotation mirrors are maintained by -the Gentoo infrastructure team. They handle the bulk of the Gentoo rsync -traffic. The community mirrors are run by volunteers from the Gentoo community. -Private mirrors are mirrors run by individuals which are closed off to the -public and meant to cut traffic costs and latency for an organization or -individual.</p> +<p> +There are three kinds of Gentoo rsync mirrors: main rotation mirrors, community +mirrors and private mirrors. Main rotation mirrors are maintained by the Gentoo +infrastructure team. They handle the bulk of the Gentoo rsync traffic. The +community mirrors are run by volunteers from the Gentoo community. Private +mirrors are mirrors run by individuals which are closed off to the public and +meant to cut traffic costs and latency for an organization or individual. +</p> -<p>At this time, we have enough community mirrors and are actively seeking +<p> +At this time, we have enough community mirrors and are actively seeking additional main rotation mirrors. Specifications for main rotation servers -include:</p> +include: +</p> <ul> <li>Minimum of a 2GHz Pentium 4 processor (or equivalent)</li> @@ -65,49 +62,57 @@ <p> These servers may be donated with bandwidth and colocation space if you have -them. Otherwise, we can provide these services and you can simply ship the +them. Otherwise, we can provide these services and you can simply ship the machine to our colocation facility. Average bandwidth consumption for a main -rotation mirror is currently ~7Mbps (around 2 TB per month). As the number of +rotation mirror is currently ~7Mbps (around 2 TB per month). As the number of main rotation mirrors increases, this number should decrease accordingly. </p> <p> If you would like to donate your machine, please email <mail -link="[EMAIL PROTECTED]">Jeffrey Forman</mail> with the pertinent information. +link="[EMAIL PROTECTED]">Jeffrey Forman</mail> with the pertinent +information. </p> </body> </section> </chapter> -<chapter> +<chapter id="local"> <title>Setting up your own local rsync mirror</title> <section> <title>Introduction</title> <body> -<p>Many users run Gentoo on several machines and need to sync the portage trees -on all of them. Using public mirrors is simply a waste of bandwidth at both -ends. Syncing only one machine against a public mirror and all others against -that computer would save resources on Gentoo mirrors and save users' bandwidth. +<p> +Many users run Gentoo on several machines and need to sync the portage trees on +all of them. Using public mirrors is simply a waste of bandwidth at both ends. +Syncing only one machine against a public mirror and all others against that +computer would save resources on Gentoo mirrors and save users' bandwidth. </p> -<p>The same holds true for organizations who would like to control the rsync -mirror their servers and workstations sync against. Of course, they usually also -want to save on badnwidth and traffic costs. </p> - -<p>All you need to do is select which machine is going to be your own local -rsync mirror and set it up. You should choose a computer that can handle the CPU -and disk load that an rsync operation require. Your local mirror also needs to -be available whenever any of your other computers syncs its portage tree. +<p> +The same holds true for organizations who would like to control the rsync +mirror their servers and workstations sync against. Of course, they usually +also want to save on bandwidth and traffic costs. +</p> + +<p> +All you need to do is select which machine is going to be your own local rsync +mirror and set it up. You should choose a computer that can handle the CPU and +disk load that an rsync operation requires. Your local mirror also needs to be +available whenever any of your other computers syncs its portage tree. Besides, it should have a static IP address or a name that always resolves to -your server. Configuring a DHCP and/or a DNS server is beyond the scope of this -guide.</p> +your server. Configuring a DHCP and/or a DNS server is beyond the scope of this +guide. +</p> -<p>Note that these instructions assume your private rsync mirror is a Gentoo +<p> +Note that these instructions assume your private rsync mirror is a Gentoo machine. If you intend to run it on a different distribution, the guide for setting up a community mirror might be more helpful. Just don't sync the mirror -every half hour but once or twice a day.</p> +every half hour but once or twice a day. +</p> </body> </section> @@ -115,11 +120,13 @@ <title>Setting up the server</title> <body> -<p>There is no extra package to install as the required software is already on -your computer.Setting up your own local rsync mirror is just a matter of -configuring the <e>rsyncd</e> daemon to make your <path>/usr/portage</path> +<p> +There is no extra package to install as the required software is already on +your computer. Setting up your own local rsync mirror is just a matter of +configuring the <c>rsyncd</c> daemon to make your <path>/usr/portage</path> directory available for syncing. Create the following -<path>/etc/rsyncd.conf</path> configuration file: </p> +<path>/etc/rsyncd.conf</path> configuration file: +</p> <pre caption="Sample /etc/rsyncd.conf"> pid file = /var/run/rsyncd.pid @@ -137,16 +144,19 @@ exclude=distfiles/ packages/ </pre> -<p> You do not need to use the <c>hosts allow</c> and <c>hosts deny</c> options. -By default, all clients will be allowed to connect. The order in which you write -the options is not relevant. The server will always check the <c>hosts allow</c> -option first and grant the connection if the connecting host matches any of the -listed patterns. The server will then check the <c>hosts deny</c> option and -refuse the connection if any match is found. Any host that does not match -anything will be granted a connection. Please read the man page (<c>man -rsyncd.conf</c>) for more information. </p> +<p> +You do not need to use the <c>hosts allow</c> and <c>hosts deny</c> options. +By default, all clients will be allowed to connect. The order in which you +write the options is not relevant. The server will always check the <c>hosts +allow</c> option first and grant the connection if the connecting host matches +any of the listed patterns. The server will then check the <c>hosts deny</c> +option and refuse the connection if any match is found. Any host that does not +match anything will be granted a connection. Please read the man page (<c>man +rsyncd.conf</c>) for more information. +</p> -<p> Now, start your rsync daemon with the following command as the root user: +<p> +Now, start your rsync daemon with the following command as the root user: </p> <pre caption="Starting the rsync daemon"> @@ -156,9 +166,11 @@ # <i>rc-update add rsyncd default</i> </pre> -<p> Let's test your rsync mirror. You do not need to try from another machine -but it would be a good idea to do so. If your server is not known by name from -all your computers, you can use its IP address instead. </p> +<p> +Let's test your rsync mirror. You do not need to try from another machine but +it would be a good idea to do so. If your server is not known by name from all +your computers, you can use its IP address instead. +</p> <pre caption="Testing your mirror"> <comment>(You may use the server name or its IP)</comment> @@ -168,14 +180,18 @@ <comment>(You should see the content of /usr/portage on your mirror)</comment> </pre> -<p> Your rsync mirror is now set up. Keep running <c>emerge --sync</c> as you -have done so far to keep your server up-to-date. If you use cron or similar +<p> +Your rsync mirror is now set up. Keep running <c>emerge --sync</c> as you have +done so far to keep your server up-to-date. If you use cron or similar facilities to sync regularly, remember to keep it down to a sensible frequency -like once or twice a day.</p> +like once or twice a day. +</p> -<note> Please note that most public mirror administrators consider syncing more -than once or twice a day an abuse. Some if not most of them will ban your IP -from their server if you start abusing their machines.</note> +<note> +Please note that most public mirror administrators consider syncing more than +once or twice a day an abuse. Some if not most of them will ban your IP from +their server if you start abusing their machines. +</note> </body> </section> @@ -183,9 +199,11 @@ <title>Configuring your clients</title> <body> -<p> Now, make your other computers use your own local rsync mirror instead of a +<p> +Now, make your other computers use your own local rsync mirror instead of a public one. Edit your <path>/etc/make.conf</path> and make the <c>SYNC</c> -variable point to your server. </p> +variable point to your server. +</p> <pre caption="Define SYNC in /etc/make.conf"> <comment>(Use your server IP addess)</comment> @@ -194,8 +212,10 @@ SYNC="rsync://<i>your_server_name</i>/gentoo-portage" </pre> -<p> You can check that your computer has been properly set up by syncing against -your own local mirror for the first time: </p> +<p> +You can check that your computer has been properly set up by syncing against +your own local mirror for the first time: +</p> <pre caption="Checking and syncing"> <comment>(Check that the SYNC variable has been setup)</comment> @@ -205,28 +225,32 @@ # <i>emerge --sync</i> </pre> -<p> That's it! All your computers will now use your local rsync mirror whenever -you run <c>emerge --sync</c>. </p> +<p> +That's it! All your computers will now use your local rsync mirror whenever you +run <c>emerge --sync</c>. +</p> </body> </section> </chapter> - <chapter> <title>Setting up a community rsync server</title> <section> <title>Introduction</title> <body> -<note> You can find sample configuration and script files in the -gentoo-rsync-mirror package. Just do <c>emerge gentoo-rsync-mirror</c> </note> +<note> +You can find sample configuration and script files in the gentoo-rsync-mirror +package. Just do <c>emerge gentoo-rsync-mirror</c>. +</note> <p> Right now, mirroring our Portage tree requires around 600Mb, so it isn't space -intensive; having at least 1Gb free should allow for growing room. Setting -up a Portage tree mirror is simple -- first, ensure that your mirror has rsync -installed. Then, set up your <path>rsyncd.conf</path> file to look something like this: +intensive; having at least 1Gb free should allow for growing room. Setting up a +Portage tree mirror is simple -- first, ensure that your mirror has rsync +installed. Then, set up your <path>rsyncd.conf</path> file to look something +like this: </p> <pre caption="rsyncd.conf"> @@ -254,24 +278,32 @@ exclude = distfiles </pre> -<p>You can pick your own locations for most of the files, of course. What's +<p> +You can pick your own locations for most of the files, of course. What's important are the section names (<c>[gentoo-portage]</c> and -<c>[gentoo-x86-portage]</c>). They are the locations that rsync clients will try -to sync from</p> +<c>[gentoo-x86-portage]</c>). They are the locations that rsync clients will +try to sync from. +</p> -<p>Above, the gentoo-x86-portage mirror points to the same data as -gentoo-portage. Although we have recently changed the official name of our -mirror to gentoo-portage, gentoo-x86-portage is still needed for backwards -compatibility, so include both entries. Eventually, the gentoo-x86-portage -location will be removed.</p> +<p> +Above, the gentoo-x86-portage mirror points to the same data as gentoo-portage. +Although we have recently changed the official name of our mirror to +gentoo-portage, gentoo-x86-portage is still needed for backwards compatibility, +so include both entries. Eventually, the gentoo-x86-portage location will be +removed. +</p> -<p> For security reasons, the use of a chrooted environment is required! This -has implications for the logged timestamps -- see the FAQ below.</p> +<p> +For security reasons, the use of a chrooted environment is required! This has +implications for the logged timestamps -- see the FAQ below. +</p> -<p> Now, you need to mirror the Gentoo Linux Portage tree. You can use the -script below to do so. Again, you'll probably want to change some of the file +<p> +Now, you need to mirror the Gentoo Linux Portage tree. You can use the script +below to do so. Again, you'll probably want to change some of the file locations to suit your needs -- in particular, they should match those of your -<path>rsyncd.conf</path>.</p> +<path>rsyncd.conf</path>. +</p> <pre caption="rsync-gentoo-portage.sh"> #!/bin/bash @@ -288,18 +320,22 @@ logger -t rsync "re-rsyncing the gentoo-portage tree" ${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1 -echo "End: "`date` >> $0.log 2>&1 +echo "End: "`date` >> $0.log 2>&1 </pre> -<p> Your <path>rsyncd.motd</path> should contain your IP address and other -relevant information about your mirror, such as information about the host -providing the Portage mirror and an administrative contact.You can now test your -server as outlined in the "Setting up your own local rsync mirror" section -above.</p> +<p> +Your <path>rsyncd.motd</path> should contain your IP address and other relevant +information about your mirror, such as information about the host providing the +Portage mirror and an administrative contact. You can now test your server as +outlined in the <uri link="#local">Setting up your own local rsync +mirror</uri> chapter above. +</p> -<p> After you have been approved as an official rsync mirror your host will be +<p> +After you have been approved as an official rsync mirror, your host will be aliased with a name of the form: <path>rsync[num].[country -code].gentoo.org</path> </p> +code].gentoo.org</path>. +</p> </body> </section> @@ -311,33 +347,41 @@ <title>Q: Who should I contact regarding rsync issues and maintenance?</title> <body> -<p> A: Visit <uri link="http://bugs.gentoo.org";>Gentoo Bugzilla</uri> and fill -out a bug on the product "Mirrors", component "Server Problem". </p> +<p> +A: Visit <uri link="http://bugs.gentoo.org";>Gentoo Bugzilla</uri> and fill out +a bug on the product "Mirrors", component "Server Problem". +</p> </body> </section> <section> <title>Q: How can I check the freshness of an official rsync server?</title> -<body> -<p>The Gentoo infrastructure team monitors all community rsync servers for -freshness. You can see the results on the <uri link="http://mirrorstats.gentoo.org/rsync";>corresponding web page</uri>. -</p></body> -</section> +<body> +<p> +The Gentoo infrastructure team monitors all community rsync servers for +freshness. You can see the results on the <uri +link="http://mirrorstats.gentoo.org/rsync";>corresponding web page</uri>. +</p> + +</body> +</section> <section> <title>Q: I run a private rsync mirror for my company. Can I still access -rsync1.us.gentoo.org?</title> +rsync1.us.gentoo.org?</title> <body> -<p> A: Because our resources are limited, we need to ensure we allocate them in -such a way as to provide the maximum amount of benefit to our users. As such, we -limit connections to our master rsync and distfile mirrors to public mirrors +<p> +A: Because our resources are limited, we need to ensure we allocate them in +such a way as to provide the maximum amount of benefit to our users. As such, +we limit connections to our master rsync and distfile mirrors to public mirrors only. Users are welcome to use our regular mirror system to establish a private rsync mirror, though they are asked to follow certain basic <uri -link="http://www.gentoo.org/news/en/gwn/20030505-newsletter.xml#doc_chap1_sect3";> -rsync etiquette guidelines</uri>. </p> +link="/news/en/gwn/20030505-newsletter.xml#doc_chap1_sect3">rsync etiquette +guidelines</uri>. +</p> </body> </section> @@ -345,8 +389,9 @@ <title>Q: Is it important that I sync my mirror twice an hour?</title> <body> -<p> A: Yes it is important. You do not need to perform the syncs at exactly :00 -and :30 but the syncs should take place in each of the following two windows: +<p> +A: Yes it is important. You do not need to perform the syncs at exactly :00 and +:30 but the syncs should take place in each of the following two windows: </p> <ol> @@ -354,14 +399,14 @@ <li>:30 to :40</li> </ol> -<p> Additionally, please make sure that your syncs are exactly 30 minutes apart. +<p> +Additionally, please make sure that your syncs are exactly 30 minutes apart. So, if you schedule the first sync of each hour for :08, please schedule the second sync of the hour for :38. </p> </body> </section> - <section> <title>Q: Where should I sync my rsync mirror before I become an official Gentoo mirror?</title> <body> @@ -379,28 +424,30 @@ <title>Q: How do I find the mirror nearest to me?</title> <body> -<p> A: <c>netselect</c> was designed to do this for you. If you haven't already -run <c>emerge netselect</c> then do it. Then run: <c>netselect +<p> +A: <c>netselect</c> was designed to do this for you. If you haven't already run +<c>emerge netselect</c> then do it. Then run: <c>netselect rsync.gentoo.org</c>. After a minute or so netselect will print an IP address. Take this address and use it as the only parameter for rsync with two colons -appended to it. eg: <c>rsync 1.2.3.4::</c>. You should be able to find out which -mirror that is from the banner message. Update your <path>/etc/make.conf</path> -accordingly. </p> +appended to it. eg: <c>rsync 1.2.3.4::</c>. You should be able to find out +which mirror that is from the banner message. Update your +<path>/etc/make.conf</path> accordingly. +</p> </body> </section> - <section> <title>Q: Can I use compression when syncing against rsync1.us.gentoo.org?</title> <body> -<p> A: No. Compression utilizes too many resources on the server, so we have +<p> +A: No. Compression utilizes too many resources on the server, so we have forcibly disabled it on <path>rsync1.us.gentoo.org</path>. Please <e>do not</e> -attempt to use compression when syncing against this server. </p> +attempt to use compression when syncing against this server. +</p> </body> </section> - <section> <title>Q: I'm seeing a lot of old and probably dead rsync processes, how can I get rid of them?</title> @@ -416,7 +463,7 @@ <pre caption="Kill old rsync processes"> /bin/kill -9 `/bin/ps --no-headers -Crsync -o etime,user,pid,command|/bin/grep nobody | \ - /bin/grep "[0-9]\{2\}:[0-9]\{2\}:" |/bin/awk '{print $3}'` + /bin/grep "[0-9]\{2\}:[0-9]\{2\}:" |/bin/awk '{print $3}'` </pre> </body> @@ -429,30 +476,30 @@ <p> In some cases, there are a few inconsiderate users who abuse the rsync mirror -system by syncing more than 1-2 times per day. In the most extreme cases, users -schedule cron jobs to sync every 15 minutes or so. This often leads to a Denial -of Service attack by continually occupying an rsync slot that could have +system by syncing more than 1-2 times per day. In the most extreme cases, +users schedule cron jobs to sync every 15 minutes or so. This often leads to a +Denial of Service attack by continually occupying an rsync slot that could have otherwise gone to another user. To try and prevent this, you may use the <uri -link="/proj/en/infrastructure/mirrors/rsyncd.conf_pl.txt">this perl script</uri> -which will scan through your rsync log files, pick out IP addresses that have -already connected more than <c>N</c> times that day and dynamically create a -<path>rsyncd.conf</path> file, including the offending IP addresses in the +link="/proj/en/infrastructure/mirrors/rsyncd.conf_pl.txt">this perl +script</uri> which will scan your rsync log files, pick out IP addresses that +have already connected more than <c>N</c> times that day and dynamically create +a <path>rsyncd.conf</path> file, including the offending IP addresses in the 'hosts deny' directive. The following line controls what <c>N</c> equals (in -this case 4): </p> +this case 4): +</p> <pre caption="Define maximum number of connections per IP"> @badhosts=grep {$hash{$_}>4} keys %hash; </pre> -<p> If you use this script, please remember to rotate your rsync log files daily +<p> +If you use this script, please remember to rotate your rsync log files daily and modify the script to match the location of your <path>rsyncd.conf</path> file. This script is tested on Gentoo Linux, but should work suitably on other -arches that support both rsync and perl. </p> +arches that support both rsync and perl. +</p> </body> </section> </chapter> - - - </guide> -- [email protected] mailing list
