Re: are partial files always deleted on interruption?
On Fri, Jul 03, 2009 at 03:32:29PM -0300, Carlos Carvalho wrote: > Both processes were killed with the same command, kill pid1 pid2. A SIGTERM (the default kill signal) should be caught by rsync, allowing it to cleanup before it exits. > However I've found that a file was left after interruption, with a > zero size. What kind of file? A temp file, or the destination file? You didn't use --inplace, right? Rsync 2.6.8 had a bug where it could leave a temp file when interrupted, but I don't see any such problem with 3.0.6. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: what happens with .~tmp~ after an interruption?
On Fri, Jul 03, 2009 at 05:25:18PM -0300, Carlos Carvalho wrote: > What happens if [some of] these files no longer exist in the source > when the next run happens? Are they, and .~tmp~, removed or kept? The file is only removed if a matching filename is transfered in the parent directory. This is because rsync doesn't know if the file was left by some other rsync that is going to still need it, or if it should clean it up. It might be nice to have rsync remove any files with a ctime older than a certain span of time, but that is not currently done. If your rsync runs are leaving .~tmp~ dirs lying around, you can either schedule your own aging of files in .~tmp~ dirs (e.g. via find), or use some custom rsync command that will only remove a .~tmp~ dir and its contents from a remote directory (e.g. via include/exclude rules). An example find command: find . -path '*/.~tmp~/*' -cmin 240 -delete That's leaves rsync to cleanup the empty dirs. You could use a more complex command to clean those too: find . -depth -path '*/.~tmp~/*' -cmin 240 -delete -o -name '.~tmp~' -cmin 240 -delete ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug? backup_dir sometimes reported on STDOUT, sometimes not.
On Fri, Jul 03, 2009 at 10:46:13PM +0200, Axel Essbaum wrote:
> In the second case the "backup_dir" report in the output is apparently
> not on STDOUT.
In all versions of rsync prior to 3.1.0 (which is not yet released), the
"backup_dir is ..." message is output very early in the transfer, so a
remote rsync cannot send it over the protocol (since it is not yet setup
right for messages). It instead sends it via STDERR (since STDOUT is
being used to talk to the client rsync). 3.1.0 fixes this by moving
this output later in the sequence of events, allowing the message to
come through the protocol and go to STDOUT as expected.
You have several options:
- Don't use double verbosity, and the message won't be output at all.
If you want to see unchanged files, you can try -ii as an alternative
way to get that info (though it does show even attribute changes, and
you'd need to be able to parse the new output format).
- Use 3.1.0 -- it's in a stable state, and I'm hoping to get it
released before too long. (I use it for all my rsyncing.)
- Patch the remote rsync to move that output from options.c to main.c.
(See the attached patch.)
..wayne..
--- a/main.c
+++ b/main.c
@@ -76,7 +76,9 @@ extern char *rsync_path;
extern char *shell_cmd;
extern char *batch_name;
extern char *password_file;
+extern char *backup_dir;
extern char curr_dir[MAXPATHLEN];
+extern char backup_dir_buf[MAXPATHLEN];
extern struct file_list *first_flist;
extern struct filter_list_struct daemon_filter_list;
@@ -752,6 +754,9 @@ static int do_recv(int f_in, int f_out, char *local_name)
exit_cleanup(RERR_IPC);
}
+ if (backup_dir && verbose > 1)
+ rprintf(FINFO, "backup_dir is %s\n", backup_dir_buf);
+
io_flush(NORMAL_FLUSH);
if ((pid = do_fork()) == -1) {
--- a/options.c
+++ b/options.c
@@ -1504,8 +1504,6 @@ int parse_arguments(int *argc_p, const char ***argv_p)
backup_dir_buf[backup_dir_len++] = '/';
backup_dir_buf[backup_dir_len] = '\0';
}
- if (verbose > 1 && !am_sender)
- rprintf(FINFO, "backup_dir is %s\n", backup_dir_buf);
} else if (!backup_suffix_len && (!am_server || !am_sender)) {
snprintf(err_buf, sizeof err_buf,
"--suffix cannot be a null string without --backup-dir\n");
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: sync performance falls off a cliff
Leen Besselink (l...@consolejunky.net) wrote on 30 June 2009 09:05: >Mike Connell wrote: >> Hi, >> > >Hi Mike, > >> I've got identical servers. One is primary the other is backup >> receiving rsyncs from the primary. I'm backing up a file system to >> disk and the files are small and there are lots of directories. >> >> The overall problem seems to be the total number of files. >> When I had ~375,000 files, the total rsync time was under a minute. >> With ~425,000 files, the total rsync time is 10 minutes. >> >> Last Friday when we were at 425,000 files, the rsync time was 10 minutes. >> Today I was able to delete 50,000 unneeded files and the rsync time went >> back down to under a minute. >> >> So why the huge change in total rsync time for a somewhat small change >> in total number of files? I'm afraid that as the total number of files keeps >> increasing that the total rsync time is going to go exponential. >> >> I turn the --progress flag on, and the time is rougly divided up evenly >> between >> building the file list and looking thru the file list. The files themselves >> are really small (~16K) and I'm not seeing any problem with anything >> other than how long it takes rsync to make a pass thru all the files. I >> do use >> the --delete option. >> >> The servers are Dell 2950s, builtin RAID 10 disks and 4Gig of RAM. >> OS is Centos 5.1. I'm running rsync 2.6.8 protocol version 29. >> >> This smells to me like some sort of caching problem. Is there something >> in the kernel or rsync itself that I can tweek? >> > >I'm no expert, but I suggest using rsync 3.x (3.0.6 for example), it >doesn't keep the as much information of the filelist in memory. Yes. Or at lease it starts transfers much faster, because it doesn't wait for the full list to be completed. >It's probably swapping to disk, because of the large list and that >significantly slows down the performance of the whole machine(s). He's probably running out of ram, not only because of rsync but also everything else. Since inodes and files are not in ram, they have to be fetched from the disk, which is *very* slow. You can tell the kernel to increase the priority of inodes, which will reduce the time to build the file list a lot. Just set /proc/sys/vm/vfs_cache_pressure to a low value. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync getting stuck while "building file list"
Hello Samba Team, We are using rysnc utility for transferring ~90GB data across the machines. We noticed that for small folders, rsync is working fine but the moment we switch to 90GB folder, it gets stuck at "building file list". We even left it running overnight e.g. 16 hours but it never came out. We are using simple rsync syntax #rsync -av --delete /data1 server2:/data RPM version 2.68-3.1 Looking forward to your advice wrt to resolution or any other details required. Thanks in advance. Regards, - Waterkingdom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: are partial files always deleted on interruption?
Wayne Davison (way...@samba.org) wrote on 4 July 2009 08:53: >On Fri, Jul 03, 2009 at 03:32:29PM -0300, Carlos Carvalho wrote: >> Both processes were killed with the same command, kill pid1 pid2. > >A SIGTERM (the default kill signal) should be caught by rsync, allowing >it to cleanup before it exits. Yes, that's why I gave the explicit kill command, showing it's a SIGTERM. >> However I've found that a file was left after interruption, with a >> zero size. > >What kind of file? A temp file, or the destination file? A temp one, like .. >You didn't use --inplace No. >Rsync 2.6.8 had a bug where it could leave a temp file when >interrupted, but I don't see any such problem with 3.0.6. That's why I asked. Isn't there a "leak" when the size is still zero? It's quite rare to happen and difficult to reproduce. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync getting stuck while "building file list"
Water Kingdom (waterking...@gmail.com) wrote on 3 July 2009 15:49: >We are using rysnc utility for transferring ~90GB data across the machines. >We noticed that for small folders, rsync is working fine but the moment we >switch to 90GB folder, it gets stuck at "building file list". We even left >it running overnight e.g. 16 hours but it never came out. > >We are using simple rsync syntax >#rsync -av --delete /data1 server2:/data > >RPM version 2.68-3.1 How many files are there? What happens if you run find /data1 -ls? Building the file list is a machine killer because it's very disk seek intensive. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync with spaces in source or destination path
Hi, I am trying to transfer a file that has spaces in its name. The rsync gives me below error. Am I doing anything wrong? #ls -l /tmp/test\ file -rw-rw-r-- 1 xxx xxx0 Jul 5 02:23 /tmp//test file # /usr/local/bin/rsync --archive /u/masanip/ACH/test\\\ file /tmp/mydir/ rsync: link_stat "/tmp/test\ file" failed: No such file or directory (2) Number of files: 0 Number of files transferred: 0 Total file size: 0 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 9 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 121 Total bytes received: 12 sent 121 bytes received 12 bytes 29.56 bytes/sec total size is 0 speedup is 0.00 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6] Thanks. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync with spaces in source or destination path
Please note that double quates is working but I want to make it work with "\\\". For example, ==> /usr/local/bin/rsync --archive "/u/masanip/ACH/test file" /tmp/mydir/ <== is working but ==> /usr/local/bin/rsync --archive /u/masanip/ACH/test\\\ file /tmp/mydir/ <== not. Thanks. On Sun, Jul 5, 2009 at 2:29 AM, paresh masani wrote: > Hi, I am trying to transfer a file that has spaces in its name. The rsync > gives me below error. Am I doing anything wrong? > > #ls -l /tmp/test\ file > -rw-rw-r-- 1 xxx xxx0 Jul 5 02:23 /tmp//test file > > # /usr/local/bin/rsync --archive /u/masanip/ACH/test\\\ file /tmp/mydir/ > rsync: link_stat "/tmp/test\ file" failed: No such file or directory (2) > > Number of files: 0 > Number of files transferred: 0 > Total file size: 0 bytes > Total transferred file size: 0 bytes > Literal data: 0 bytes > Matched data: 0 bytes > File list size: 9 > File list generation time: 0.001 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 121 > Total bytes received: 12 > > sent 121 bytes received 12 bytes 29.56 bytes/sec > total size is 0 speedup is 0.00 > rsync error: some files/attrs were not transferred (see previous errors) > (code 23) at main.c(1039) [sender=3.0.6] > > Thanks. > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync with spaces in source or destination path
On 05.07.2009 02:29, paresh masani wrote: > Hi, I am trying to transfer a file that has spaces in its name. The rsync > gives me below error. Am I doing anything wrong? What you are looking for is "-s" a.k.a. "--protect-args' - man rsync - -s, --protect-args no space-splitting; wildcard chars only - snip - Bis denn -- Real Programmers consider "what you see is what you get" to be just as bad a concept in Text Editors as it is in women. No, the Real Programmer wants a "you asked for it, you got it" text editor -- complicated, cryptic, powerful, unforgiving, dangerous. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync with spaces in source or destination path
Got it work with single slash. :-) # /usr/local/bin/rsync --archive /u/masanip/ACH/test\ file /tmp/mydir/ On Sun, Jul 5, 2009 at 2:34 AM, paresh masani wrote: > Please note that double quates is working but I want to make it work with > "\\\". > > For example, ==> /usr/local/bin/rsync --archive "/u/masanip/ACH/test file" > /tmp/mydir/ <== is working but ==> /usr/local/bin/rsync --archive > /u/masanip/ACH/test\\\ file /tmp/mydir/ <== not. > > Thanks. > > > On Sun, Jul 5, 2009 at 2:29 AM, paresh masani wrote: > >> Hi, I am trying to transfer a file that has spaces in its name. The rsync >> gives me below error. Am I doing anything wrong? >> >> #ls -l /tmp/test\ file >> -rw-rw-r-- 1 xxx xxx0 Jul 5 02:23 /tmp//test file >> >> # /usr/local/bin/rsync --archive /u/masanip/ACH/test\\\ file /tmp/mydir/ >> rsync: link_stat "/tmp/test\ file" failed: No such file or directory (2) >> >> Number of files: 0 >> Number of files transferred: 0 >> Total file size: 0 bytes >> Total transferred file size: 0 bytes >> Literal data: 0 bytes >> Matched data: 0 bytes >> File list size: 9 >> File list generation time: 0.001 seconds >> File list transfer time: 0.000 seconds >> Total bytes sent: 121 >> Total bytes received: 12 >> >> sent 121 bytes received 12 bytes 29.56 bytes/sec >> total size is 0 speedup is 0.00 >> rsync error: some files/attrs were not transferred (see previous errors) >> (code 23) at main.c(1039) [sender=3.0.6] >> >> Thanks. >> > > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync with spaces in source or destination path
Hi, On Sun, 5 Jul 2009, paresh masani wrote: Hi, I am trying to transfer a file that has spaces in its name. The rsync gives me below error. Am I doing anything wrong? #ls -l /tmp/test\ file -rw-rw-r-- 1 xxx xxx0 Jul 5 02:23 /tmp//test file # /usr/local/bin/rsync --archive /u/masanip/ACH/test\\\ file /tmp/mydir/ rsync: link_stat "/tmp/test\ file" failed: No such file or directory (2) Don't you see it: one / too much. Viele Gruesse Eberhard Moenkeberg (emoe...@gwdg.de, e...@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoe...@gwdg.de Tel.: +49 (0)551 201-1551 - Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL:http://www.gwdg.de E-Mail: g...@gwdg.de Tel.: +49 (0)551 201-1510Fax:+49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Bernhard Neumair Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 - -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
