Re: rsync in-place (was Re: rsync 1tb+ each day)
On Wed, 5 Feb 2003, Ben Escoto wrote: > > "CB" == Craig Barratt <[EMAIL PROTECTED]> > > wrote the following on Wed, 05 Feb 2003 04:41:22 -0800 > > CB> Of course, a major issue with --inplace is that the file will be > CB> in an intermediate state if rsync is killed mid-transfer. Rsync > CB> currently ensures that every file is either the original or new. > > I'm curious, how does it ensure this? During the copy, rsync writes to a temporary file in the same directory (the temp file is hidden; it starts with a "."). Then, once the transfer is done, it mv's that temp file over the original. My understanding is that mv is atomic under unix, so this action either happens in its entirety or not at all. Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Feature request...
Hi Dan, > I would like a '-i, --interactive' option > to use with --delete, so I can decide wich > deletes to up date and which were potential > mistakes. It sounds like you might be interested in Unison: http://www.cis.upenn.edu/~bcpierce/unison/ Another option would be to rsync with a hard-link copy. This way you'd get what appear to be two separate filesystem trees, but only take up the space of the original + differences, and a list of the differences (by running rsync with -v). You could select which files to keep at that point and then erase the "duplicate" tree. This is discussed in the context of incremental backups at: http://www.mikerubel.org/computers/rsync_snapshots/ If you go this route, be sure to use J.W.'s new --link-dest option as it keeps the permissions straight. Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Help With Restoring
> rsync --delete --stats --compress --recursive --times --perms --links > --rsh=/usr/bin/ssh --exclude "tmp/" --exclude "dev/" --exclude "proc/" > --exclude "backups/ " --delete-excluded --backup > --backup-dir=/backup2/BACKED_UP_SERVER_FQDN/$DAY -a /* > CENTRAL_SERVER_IP:/backup2/BACKED_UP_SERVER_FQDN/current && echo "Daily > backup ran on `date`" >> /var/log/backup.log Notice that if you have any dotfiles in / (that is, flies that begin with a "."), this will not copy them, but normally you wouldn't have dotfiles in / so it shouldn't matter. Might want to start using / instead of /* from now on though, just for good form. > This has been working fine and I've even been able to restore files using > scp from time to time. now I'm faced with a bare metal recovery cause of a > botched upgrade from Redhat 7.1 to 8.0 that failed half way through. This > machine was still accessible via console and ssh (putty). I signed in via > ssh and ran... > > # cd / > # scp -r CENTRAL_SERVER_IP:/backup1/BACKED_UP_SERVER_FQDN/current/.* . > > This was running for a while and then i lost my connection and cant > reconnect. I won't have console access till the morning but any advice > would be greatly appreciated. Don't you mean: scp -rp CENTRAL_SERVER_IP:/backup1/BACKED_UP_SERVER_FQDN/current/. . ? The command you list above won't copy anything, since there are no dotfiles in the root directory. Also, you have backup2 in the rsync command and backup1 in the scp command. Maybe that was just a typo though... One final thought. Sometimes scp has issues with symlinks (as in, copying the target instead of the link). Might want to use: cd / rsync -av CENTRAL_SERVER_IP:/backup1/BACKED_UP_SERVER_FQDN/current/ . Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync windows -> unix still hanging :(
On Tue, 7 Jan 2003, Dave Dykstra wrote: > I've never seen anybody suggest any other value for --modify-window than "2". > Does 90 really work better than 2? My understanding is that PC-style > filesystems do not have 1 second timestamp granularity and so round the > times up or down slightly. Dave, Tim informed me (off list) that --modify-window=10 also seemed to work. Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: HELP! rsync won't upload my .htpasswd files
> I'm using rsync to synchronize between our staging server and our > production server. All of my files are uploading correctly except for a > couple of .htpasswd files which reside in /var/apache/webscripts > > I'm guessing that they won't upload because they start with "." Is there > some way around this? It's very important that these files get uploaded > to this directory. Damon, Change: /var/apache/webscript/* to /var/apache/webscript/ (including the trailing slash). Mike ps> your email client seems to be sending html email; might want to get rid of that... -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync windows -> unix still hanging :(
> I think I got it. > > It must have something to do with the file modification times and perhaps > the number of places of accuracy, perhaps in fractions of a second. > > When I add --modify-window=90 as a parameter. Then it works as it should. > > At least it does, for me now. You might try a smaller number (of seconds) > but for me. 90 seconds is close enough! Timothy, Aha! I'll add this to the bugs:windows section of my backup page, http://www.mikerubel.org/computers/rsync_snapshots/ May I cite your email verbatim? Thanks, Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync windows -> unix still hanging :(
On Fri, 3 Jan 2003, Timothy Burt wrote: > I am seeing similar symptoms. Hi Timothy, I'm trying to find out more about this. May I ask, are the files you see sent even when they haven't changed regular files, or directories, or both? Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync windows -> unix still hanging :(
On Sat, 28 Dec 2002, Scott Evans wrote: > I spent about 12 hours today writing a nice backup system based on > Mike Rubel's snapshot system; but much to my dismay, when I took it > for a test drive, I found that rsync hung on the first large directory > I tried to back up. Aiee! [ ... ] Hey Scott, I'm Mike Rubel (author of that snapshot system page)--I haven't heard about this cygwin issue before, but this would definitely be worth adding to the FAQ. I'm sorry to hear you're having so much trouble with it! You're actually the second person to mention windows issues; the other fellow was using a SAMBA mount and rsyncing locally on the server. The problem there was files mysteriously getting copied when they hadn't changed, or deleted when they weren't supposed to be. I asked the list if anyone else had seen that behavior, but no one responded, so I'm not sure what to think. I'll add it to the FAQ, anyway, and maybe the BUGS section. Please let me know if you find any additional information. I'm on the rsync list but not the cygwin list (yet). I've had no trouble with this at home with my roommate's Win2K PC, but between this case and the previous one, it sounds like the rsync/windows interaction may still have some rough edges, which is not really acceptable for a backup solution. :( Thanks, Mike ps> I'm moving the site to a new server tonight; I tested it manually and it appears to be in order, but DNS probably won't catch up until tomorrow morning. Please let me know if you run into any trouble connecting! -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync snapshot problem deleting files
> Well, I followed this link: > http://www.mikerubel.org/computers/rsync_snapshots/ > > and implemented it (with a bit of tweaking) on my home network. It is > used to make snapshots of two Windows PC's over samba. On one of the > machines (and not the other, from what I can tell) it deletes files from > the target machine for no good reason (ie. they still exist on the > source machine). Then, if I run it again right away, it gets the files anew. Hi again all, Has anyone else seen Mark's problem, which sounds like a weird interaction issue between samba and rsync? Mark, I will add a note about your experience here to the web page; so far it's the only report I've seen though, and I'd really like to get more data. Is anyone else running the rsync-snapshot trick from windows machines via Samba? Can we get more debugging information to determine the source of the problem? Thanks, Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync snapshot problem deleting files
> Well, I followed this link: > http://www.mikerubel.org/computers/rsync_snapshots/ > > and implemented it (with a bit of tweaking) on my home network. It is > used to make snapshots of two Windows PC's over samba. On one of the > machines (and not the other, from what I can tell) it deletes files from > the target machine for no good reason (ie. they still exist on the > source machine). Then, if I run it again right away, it gets the files anew. Hi Mark, I'm glad you were able to get the snapshot trick going on your home network, though I'm sorry to hear about this trouble with the deletes. I studied the situation under Samba for quite a while on my system before proclaming it working, so this is interesting and a bit worrying. The only irregularity I noticed on my systems was that sometimes files would get sent over even if they hadn't changed, which I assumed was a peculiarity of Samba, since it never happened from my Linux machines. It was only an extra file-send or two every few sync's, so I didn't worry about it too much at the time. Has anyone else tried what Mark describes here? Are there any suggestions or comments? I'll stay tuned for a couple of days, and then make a note on the website, since this is something other potential implementers will need to know about. Mike Rubel -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Feature Request: break hardlinks before metadata changes
Hi Tripp, Glad you saw J.W.'s patch; this business of unlinking before a metadata change would be nice, but I'm not sure how to do it without introducing yet another patch! Best regards, Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: restricting rsync over ssh
> Somewhat belatedly, I can report that I use rysnc in daemon mode in > conjunction with an SSH tunnel, but using remote port forwarding. I > use the method to distribute password and shadow files. It seemed like this ought to be possible. So, out of curiousity, why does rsync include a "-e ssh" option at all? Wouldn't it be simpler to pick one protocol (rsync) and let the user tunnel it over his or her favorite encrypted transport? Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsyncing read-only files
> I have two machines, call then A and B. I'm using rsync on B to download > from A (I'm running rsync in daemon mode on A from inetd.conf). The rsyncing > on B works fine except for one problem: when I run rsync on B to do the > downloading from A, I get permission denied for files on A that have mode > 600. > > Is there a way to get rsync to download all files, irrespective of their > modes? Hi Stewart, I may be wrong, but this sounds like it might be a file permissions problem and not an rsync problem. Files that have permission 600 can be read and written by their owner, but no-one else. If their owner is root, then rsyncd can only read them if it is running as root. What user is rsyncd running as now? Here's one way to find out: ps -f `pidof rsyncd` If this is indeed your problem, you have a couple of options: 1) run rsyncd as that user (works, but potential security issue) 2) find a new solution Hope that helps-- Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: restricting rsync over ssh
> > If so, I am trying to find the best way to restrict rsync -e ssh on the > > remote machine. Prepending the authorized_keys entry with > > command='rsync ...' 1024... results in the 'Protocol mismatch - is your > > shell clean?' error. This brings up an interesting question. Does anyone use the server version of rsyncd with an ssh tunnel? In other words: On the server, bring up sshd listening on *:22, and rsyncd accepting connections from localhost only, port 873. Then, on the client machine, set up an ssh tunnel: ssh server -L 1873:server:873 (I just picked 1873 for convenience; you could use any open user-accessible port). Then rsync to localhost:1873, letting ssh carry the connection over the tunnel to the server: rsync rsync://user@localhost:1873/path/to/source dest Does anyone use this approach? Are there any advantages/disadvantages? Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync as a backup solution
Eric, I'm going to use this opportunity to shamelessly plug my website: http://www.mikerubel.org/computers/rsync_snapshots/ By keeping multiple rotating backup "snapshots", you can protect yourself should one or more of them get hosed, and you still won't need a huge amount of extra storage. All you need are rsync and cp. Mike > I would like to get some opinions on a backup method using rsync. I already > have a script written to mirror the data over to a backup script from a cron > that is run every night. I've also been tempted to use the "--delete" > option to keep everything up to date on the backup server. However, this is > probably a bad idea in the event my data is deleted from the "/home" > directory (just as an example) before the cron runs. When the cron runs and > this isn't noticed before hand, the "/home" directory on the backup server > will probably be whiped out as well, correct? So, do all you that use rsync > as a backup method not use the "--delete" option, or do you keep maybe a > separate weekly backup safe somewhere else. Of course, we do have our data > center making tape backups of the backup server as well, but let's assume we > didn't. Thank you! -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: possibly new rsync trick: rotating snapshots
mr> Quite right--I didn't mean to imply that NFS to a remote location (or mr> even across a firewall from a DMZ) is a good idea; I would use rsync mr> over ssh for that, or rsyncd as you suggest. The NFS idea was more mr> for a dedicated "snapshot server", where the main file server would be mr> mirrored directly across a dedicated subnet. mp> Even for a local network, you will probably find rsyncd substantially mp> faster than NFS. (rsync can more intelligently pipeline and overlap mp> operations on its own socket than the OS can for NFS.) Certainly it's mp> nice to also support NFS for direct access though. Martin, Thanks for pointing this out to me--I've added a note on the web page near the NFS discussion. My machines are so slow that the network isn't a bottleneck anyway, but this should be relavent to others, especially where users don't need the transparency of a mounted system directory. http://www.mikerubel.org/computers/rsync_snapshots.html Best regards, Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: why --delete dosen't work in my script?
> In my backup script I have noticed nothing gets deleted on the remote > end even though I have --delete and --delete-excluded. Did I miss > anything obvious here? Not sure if this applies to you, but... One thing I noticed about trying to use --delete in combination with -b --backup-dir is that files do not get deleted if there is already another hard link to them on the target side. As I understand it, this is not so much a bug in rsync as it is a somewhat non-intuitive POSIX behavior in my operating system (Linux). Basically, if it can, backup just renames the file instead of copying it. But on my system, if a and b are both hard links to the same inode, mv a b does *not* unlink a! It has no effect, in fact. My solution was to make two rsync passes, the first with -b --backup-dir but no --delete, the second without -b --backup-dir but with --delete. At some point, though, maybe we should patch rsync to make it do the intuitive thing in this case. Does anyone else have experience with this? Regards, Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: possibly new rsync trick: rotating snapshots
> In addition to the comments regarding NFS etc. You will > find that with cp -l permissions and ownership will not be > accurately preserved in older snapshots if someone chmod or > chowns files on the original. Oh--Thank you, you're quite right, I missed that. If you use rsync -a, then rsync *will* change the ownership/permissions on the target file, but will *not* first unlink it (I had been working on the assumption that rsync always behaves as if it had a --remove-destination flag set, but it looks like that only holds for data, not metadata). * To summarize: if a file's permissions/ownerships are changed in place, * then on the next snapshot the new permissions/ownerships will affect all * previous snapshots, not just the most recent one. This is a serious * problem in my mind, and I apologize for missing it. I'm looking into a workaround now; in the meantime, I posted an update at the top of the page. > I have a complete system for automated rotating backups that > i was planning on announcing shortly (GPL). I have been > using it for about a month so far and am quite pleased. > It is a little more complex than your approach and provides > more flexible rotation schedules with automatic image > (snapshot) expiration. > > If anyone wishes to look at it or be an alpha test site > a periodically updated snapshot of the development area can > be found at http://www.pegasys.ws/dirvish. Looks interesting! I'd like to give it a try. Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: possibly new rsync trick: rotating snapshots
> That looks good, particularly the description of `cp -al'. I'll add a > link. Hi Martin, First of all, thanks *so* much for all of your work on rsync. It's a really terrific tool, and I seem to keep finding new uses. :) > I would use rsync's daemon mode, rather than NFS, to do backups to a > remote server. It's equally secure and much more efficient. > > Similarly, if my user base was sufficiently technical to use rsync to > retrieve files, then I'd just export the backups that way. I guess > this is only useful in a situation where you trust the users not to > read each other's files. Quite right--I didn't mean to imply that NFS to a remote location (or even across a firewall from a DMZ) is a good idea; I would use rsync over ssh for that, or rsyncd as you suggest. The NFS idea was more for a dedicated "snapshot server", where the main file server would be mirrored directly across a dedicated subnet. The advantages of using that approach would be transparency for users (the snapshots would just look like an ordinary directory on the file server) and protection from server root compromise. I'll try to clarify that on the site. Thanks again, Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
possibly new rsync trick: rotating snapshots
Hello all, I've found a simple, rsync-based trick to emulate the "rotating snapshots" feature found on some file servers, and hope that members of this group will find it useful and/or interesting: http://www.mikerubel.org/computers/rsync_snapshots/ Snapshots are (or rather, appear to be) full backups of a system taken at regular intervals, available directly to users through a special read-only system directory. When users accidentally delete or overwrite important files, or find themselves in need of an old version, they can copy from the snapshots without any sysadmin help. Since I first saw the labor savings that automatic, rotating snapshots provide (relative to restore from incremental tape backups), I've been searching for a way to mimic them on my home Linux machine. Rsync gave me a way to deal with the difficult part--making what are really incremental backups appear to be full ones. Thanks in advance for any feedback or suggestions-- Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Incremental backups and batch mode.
Diego wrote: > Right, wonderful, but let's consider a big database file, let's say > a 2Gbyte file, that is slightly changed every day of about a 10% ... > So at every backup the whole 2Gbyte file is saved. ... > So I would like to use the rsync algorithm to calculate the differences > (delta files) for the levels n>0 in the same order dump and tar work > but saving much more tape space. Ah... now I see. Unfortunately, this one's over my head. Can anyone else help here? Can rsync deal explicitly with parts of files? Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Incremental backups and batch mode.
> Something similar: > I would like to have a first snapshot (level 0) that is a complete copy, > and then other incremental backups that are just delta files > (just the differences from the level 0 snapshot). The "normal" utilities for this job would be dump and tar, especially if you're dumping to tape. You can also use rsync, but it's somewhat indirect if you're dumping to tape! :) > That should be done saving the checksums of the level 0 backup locally > and then checking the current files against those checksums to calculate the > delta files to be saved as leve 1 backup, and so on. Okay, one thing that takes a little getting used to here is that if you use rsync, the backup order is reversed. Let's see if I can explain it here. Using dump or tar, the 0 backup is large; it contains the whole filesystem at the time it was made. Then the 1 backup is smaller; it contains only the changes made between t_0 and t_1. The 2 backup would also be small, consisting only of changes between t_1 and t_2. And so on. Using rsync, the process is reversed. The *most recent* backup is the big one, and *earlier* backups contain only the files that changed. So the 0 backup is the most recent one, the 1 backup contains only those files that changed between t_1 and the most recent backup; the 2 backup contains only those files that changed between t_2 and t_1, and so on back in time. It's counterintuitive, but it's vastly more efficient for remote backups since you only need to do a full dump once, then never again. Now, how would you implement it? For simplicity's sake, I'm going to say that you're backing up /home into the directory /home-backup. Extending that to backup on a remote machine is a separate (albeit easy) issue, so I won't cover that here. Under /home-backup, you make folders like so (you'll probably find your own names for these folders): /home-backup/current/ /home-backup/current-1day/ /home-backup/current-2day/ The idea is that current/ would contain the current image (most of the files), current-1day/ would contain only the files that changed since yesterday, and current-2day/ would have anything that changed between two days ago and yesterday. You can have as many of these as you want, and they don't have to be evenly spaced; this is just for example. Now, to make it work, run something like this once a day: # delete the oldest incremental backup rm -rf /home-backup/current-2day # shift the intermediate incremental backups back by one mv /home-backup/current-1day /home-backup/current-2day # rsync into /home-backup/current, copying any changed files into the # folder current-1day first rsync -vab --delete --backup-dir=/home-backup/current-1day /home/ \ /home-backup/current You can also use exclude lists and all that other stuff. Is this clear? Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
rsync-2.5.4:backup.c possible bug
Hello, I suspect a minor bug in the file rsync-2.5.4/backup.c line 206. To identify too-long pathnames, it makes the following test: if (strlen(backup_dir) + strlen(fname) > (MAXPATHLEN - 1)) I think that the constant subtracted off should be 2, to account both for the forward-slash and for the trailing zero in the following snprintf statement, line 211: snprintf(keep_name, sizeof (keep_name), "%s/%s", backup_dir, fname); Notice that (line 178) keep_name is an array of MAXPATHLEN characters. Another (perhaps more logical) possible fix would be to make keep_name MAXPATHLEN+1 characters long. It depends whether MAXPATHLEN was intended to count the trailing zero or not. On my linux machine, MAXPATHLEN is 4095, so I assume that it was not. I sent this to the web bug-submission form about a week and a half ago, but haven't heard anything since then, so I'm forwarding it on to the list here. Please let me know if there's a more appropriate place to send it! Thanks, Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Excludes not working
Bart Brashers wrote: > ... I mount them using samba (to avoid having to install the > cygwin version on each box) then run (using rsync 2.5.2 run from crontab) > > rsync -vuaz --delete-excluded --exclude-from=/root/bin/rsync.exclude \ >--modify-window=2 /mnt/pc/machine/share /backup/machine/share > > It works like a charm, except for some excludes that won't work -- rsync > still copies the files. Here's my exclude file: > /mnt/pc/machine/D/Music ... Hi Bart, I'm not an expert with rsync, so someone please correct me if I've got this wrong, but I think the way to exclude /mnt/pc/machine/D/Music would be like this: /Music/ Otherwise it's looking to exclude: /mnt/pc/machine/share/mnt/pc/machine/D/Music, which of course doesn't exist. Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
