Re: Writing to the wrong directory
created directory /media/2tb/bak/da The cron job was too long and truncated. I guess last time it truncated at /media/2tb/ Any guesses on where the command length limit is? Cron? bash? It sure would be nice for cron to spit out an error if I create a job that's too long. -- I don't want people who want to dance, I want people who have to dance. --George Balanchine http://www.ChaosReigns.com -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Writing to the wrong directory
I expect this is user error, but I thought I'd post in case anyone else is feeling a similar sense of losing their mind. I have a root cron job: 0 3 * * * rsync -Hva --stats --del ... / /media/2tb/bak/dancer-`date +\%F`/ ... represents 4 --link-dest's and 22 --exclude's (including /media/). So last night, / should have been copied to /media/2tb/bak/dancer-2010-11-29/. Instead, when I checked on my backups this morning, instead of /media/2tb/bak/dancer-2010-11-29/, / had been copied to /media/2tb/. The differences corresponded exactly to the --excludes. There was no /media/2tb/dev/, and /media/2tb/tmp/ contained what it had the previous day, not the same as /tmp/. And of course, due to the --del, it deleted everything else I had in /media/2tb/. I have since removed the --del, since it was there for reasons no longer relevant. /media/2tb is, as you might imagine, a mount point for a 2tb hard drive. The only way I can imagine this happening is if /media/2tb/bak/dancer-2010-11-29/ were a simlink to /media/2tb/. Which is madness. Or I guess I could have run it from the command line, but: dar...@dancer:~$ history | grep rsync 3471 man rsync 4997 history | grep rsync | less 4999 history | grep sudo.*rsync 5004 history | grep rsync dar...@dancer:~$ root's history contains rsync commands with the targets: /media/2tb/bak/dancer-`date +%F`/ dancer-2010-11-01b dancer-2010-11-01b /media/2tb/bak/dancer-`date +%F`/ I did verify the copy of / in /media/2tb/ contained an update to a file I made just last night. Since I used the -a flag to rsync, I don't think it's possible to verify the copy happened around the time of the cron job. Any other ideas? Other than I or someone else, unknown to me, maliciously kicked off this rsync, and then removed the evidence? -- I don't want people who want to dance, I want people who have to dance. --George Balanchine http://www.ChaosReigns.com -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: apache log backups
On 01/09, Wayne Davison wrote: example, www.chaosreigns.com-access.log.196.gz on the origin is the same file as www.chaosreigns.com-access.log.186.gz on the destination, so The --fuzzy option might help, but only if the filenames that moved don't already exist. Rsync expects that an existing file is the right No most of the files already exist. file to use for old data, and never checks for a different file as a better source of old data. For the case of rotated logs, you're better off first running an appropriate number of file rotations on the receiving system, and then doing a copy of the new files. Surely an external script running on the other end trying to figure out how to get the data on the destination to match the origin before rsyncing is not the ideal way to handle this. I realize handling it with checksums might be overkill. What about, anytime rsync comes across a file on the origin that includes .log.number.gz (or bz2) it looks for a file with the same size and modification time with a different number on the destination, it remembers the difference for that directory and does the rest of the comparisons in that directory for matching files with the same offset first, then with the same file name as usual (per file)? For example, it comes across www.chaosreigns.com-access.log.196.gz on the origin, and notices that that file on the destination doesn't match time and filesize, so it looks at the files in this sequence: www.chaosreigns.com-access.log.197.gz www.chaosreigns.com-access.log.195.gz www.chaosreigns.com-access.log.198.gz www.chaosreigns.com-access.log.194.gz www.chaosreigns.com-access.log.199.gz www.chaosreigns.com-access.log.193.gz www.chaosreigns.com-access.log.200.gz www.chaosreigns.com-access.log.192.gz www.chaosreigns.com-access.log.201.gz www.chaosreigns.com-access.log.191.gz www.chaosreigns.com-access.log.202.gz ... www.chaosreigns.com-access.log.187.gz www.chaosreigns.com-access.log.208.gz www.chaosreigns.com-access.log.186.gz Until it gets to www.chaosreigns.com-access.log.186.gz and notices it does match timestamp and file size, and records (only while it's in this directory) that the offset is -10. So when it looks at the file named, say, www.chaosreigns.com-access.log.39.gz it first checks www.chaosreigns.com-access.log.29.gz on the destination and if that matches, renames it to www.chaosreigns.com-access.log.29.gz. If it doesn't match it behaves as it normally would, transferring www.chaosreigns.com-access.log.39.gz from the origin. Any chance of that feature being added? These logs being unnecessarily transferred accounted for about 55% of the data in my 26 hour rsync... 14 unnecessary hours. -- Blessed are they who, in the face of death, think only about the front sight. http://www.ChaosReigns.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync -vae ssh user@host1:/tmp/dir user@host2:/tmp/
On 10/24, jw schultz wrote: No. Use ssh to set up port forwarding. If you know not how, use the ssh resources. I can't because of firewalls. workstation can connect to port 22 on both host1 and host2. host1 cannot connect to any ports on either workstation or host2. host2 cannot connect to any ports on either workstation or host1. So if I understand correctly, port forwarding cannot resolve this problem, and the only solutions are to either implement the syntax in the subject of this email in rsync, or download everything from host1 to workstation temporarily, and then upload it all to host2 ? -- I would believe only in a God that knows how to Dance. - Nietzsche http://www.ChaosReigns.com -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync -vae ssh user@host1:/tmp/dir user@host2:/tmp/
On 10/24, pod wrote: If you are able to make ssh connections from host1 to home then the method outlined in No, sorry for omitting that information, but the two remote hosts cannot connect to the home machine (my workstation, and I doubt necessary connections are allowed involving any other machines). (so ssh / nc port forwarding won't work either) host1$ rsync --rsh ssh --rsync-path 'ssh host2 rsync' srcfile home:dstfile Cute. So, since I'm not just missing something, any chance of support for rsyncing from one remote machine to another remote machine being added ? -- It's never too late to panic. http://www.ChaosReigns.com -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync -vae ssh user@host1:/tmp/dir user@host2:/tmp/
I have legitimate ssh access to two remote machines. ssh directly from either machine to the other is blocked by firewalls which I cannot control. $ rsync -vae ssh [EMAIL PROTECTED]:/tmp/dir [EMAIL PROTECTED]:/tmp/ receiving file list ... done rsync: mkdir [EMAIL PROTECTED]:/tmp: No such file or directory rsync error: error in file IO (code 11) at main.c(319) I would like this to copy host1:/tmp/dir into host2:/tmp/. Is there some way to do this that I'm missing ? Without copying the gigabytes of data to my local hard drive first before sending to the second machine (which I'm currently doing) ? (scp has a similar problem, but rsync is often more useful to me) -- You may be recognized soon. Hide. http://www.ChaosReigns.com -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
