Statistics appearing in middle of file list -- no errors
Has anybody seen this? We want to seperate the statistics out from the file list, and were using tail to grab the end of the file. the command we run is: rsync -r -a -z --partial --suffix=".backup" --exclude="*.backup" \ --stats -v /. 10.1.1.60::cds101/ > /var/log/rsync.log 2>&1 along with a number of excludes to skip the /tmp, /dev, /var and /proc directories. The output in file /var/log/rsync.log is: building file list ... done dev/ttyp0 etc/cups/certs/ etc/cups/certs/0 etc/mail/statistics root/.bash_history smb_shares/var/lib/dhcp/ smb_shares/var/lib/dhcp/dhcpd.leases smb_shares/var/lib/dhcp/dhcpd.leases~ smb_shares/var/log/debug smb_shares/var/log/mail smb_shares/var/log/messages smb_shares/var/log/rsync.log smb_shares/var/log/secure smb_shares/var/run/utmp smb_shares/var/spool/clientmqueue/ smb_shares/var/spool/mail/ smb_shares/var/spool/mail/root smb_shares/var/spool/mqueue/ usr/local/samba/var/locks/ usr/local/samba/var/locks/browse.dat Number of files: 169315 Number of files transferred: 13 Total file size: 1714847358 bytes Total transferred file size: 1013994 bytes Literal data: 30552 bytes Matched data: 983834 bytes File list size: 3438061 Total bytes written: 3442643 Total bytes read: 8794 wrote 3442643 bytes read 8794 bytes 9094.70 bytes/sec total size is 1714847358 speedup is 496.85 dev/ etc/cups/certs/ etc/mail/ root/ smb_shares/var/lib/dhcp/ smb_shares/var/log/ smb_shares/var/run/ smb_shares/var/spool/mail/ usr/local/samba/var/locks/ Any ideas? Thanks! Ed King -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Statistics appearing in middle of file list -- no errors
I thought I was -- upgraded to the latest this morning and am testing now. I was running 2.4.3. I'll let you know if this clears the problem. Thanks! Ed Craig Barratt wrote: Has anybody seen this? We want to seperate the statistics out from the file list, and were using tail to grab the end of the file. the command we run is: rsync -r -a -z --partial --suffix=".backup" --exclude="*.backup" \ --stats -v /. 10.1.1.60::cds101/ > /var/log/rsync.log 2>&1 along with a number of excludes to skip the /tmp, /dev, /var and /proc directories. The output in file /var/log/rsync.log is: building file list ... done dev/ttyp0 etc/cups/certs/ etc/cups/certs/0 etc/mail/statistics root/.bash_history smb_shares/var/lib/dhcp/ smb_shares/var/lib/dhcp/dhcpd.leases smb_shares/var/lib/dhcp/dhcpd.leases~ smb_shares/var/log/debug smb_shares/var/log/mail smb_shares/var/log/messages smb_shares/var/log/rsync.log smb_shares/var/log/secure smb_shares/var/run/utmp smb_shares/var/spool/clientmqueue/ smb_shares/var/spool/mail/ smb_shares/var/spool/mail/root smb_shares/var/spool/mqueue/ usr/local/samba/var/locks/ usr/local/samba/var/locks/browse.dat Number of files: 169315 Number of files transferred: 13 Total file size: 1714847358 bytes Total transferred file size: 1013994 bytes Literal data: 30552 bytes Matched data: 983834 bytes File list size: 3438061 Total bytes written: 3442643 Total bytes read: 8794 wrote 3442643 bytes read 8794 bytes 9094.70 bytes/sec total size is 1714847358 speedup is 496.85 dev/ etc/cups/certs/ etc/mail/ root/ smb_shares/var/lib/dhcp/ smb_shares/var/log/ smb_shares/var/run/ smb_shares/var/spool/mail/ usr/local/samba/var/locks/ Any ideas? Thanks! The final output appears to be from the final directory permission fixup. The child process on the receiving side generates the stats, then does deletes, hardlinks and a final fix of the directory mtimes. In 2.5.5 this output should be disabled, see line 290 of generator.c: /* f_out is set to -1 when doing final directory permission and modification time repair */ if (set_perms(fname,file,NULL,0) && verbose && (f_out != -1)) rprintf(FINFO,"%s/\n",fname); return; Are you running 2.5.5? Craig
Re: Statistics appearing in middle of file list -- no errors
Version 2.5.5 on the receiving side seems to fix the problem. Thanks! Ed Craig Barratt wrote: Has anybody seen this? We want to seperate the statistics out from the file list, and were using tail to grab the end of the file. the command we run is: rsync -r -a -z --partial --suffix=".backup" --exclude="*.backup" \ --stats -v /. 10.1.1.60::cds101/ > /var/log/rsync.log 2>&1 along with a number of excludes to skip the /tmp, /dev, /var and /proc directories. The output in file /var/log/rsync.log is: building file list ... done dev/ttyp0 etc/cups/certs/ etc/cups/certs/0 etc/mail/statistics root/.bash_history smb_shares/var/lib/dhcp/ smb_shares/var/lib/dhcp/dhcpd.leases smb_shares/var/lib/dhcp/dhcpd.leases~ smb_shares/var/log/debug smb_shares/var/log/mail smb_shares/var/log/messages smb_shares/var/log/rsync.log smb_shares/var/log/secure smb_shares/var/run/utmp smb_shares/var/spool/clientmqueue/ smb_shares/var/spool/mail/ smb_shares/var/spool/mail/root smb_shares/var/spool/mqueue/ usr/local/samba/var/locks/ usr/local/samba/var/locks/browse.dat Number of files: 169315 Number of files transferred: 13 Total file size: 1714847358 bytes Total transferred file size: 1013994 bytes Literal data: 30552 bytes Matched data: 983834 bytes File list size: 3438061 Total bytes written: 3442643 Total bytes read: 8794 wrote 3442643 bytes read 8794 bytes 9094.70 bytes/sec total size is 1714847358 speedup is 496.85 dev/ etc/cups/certs/ etc/mail/ root/ smb_shares/var/lib/dhcp/ smb_shares/var/log/ smb_shares/var/run/ smb_shares/var/spool/mail/ usr/local/samba/var/locks/ Any ideas? Thanks! The final output appears to be from the final directory permission fixup. The child process on the receiving side generates the stats, then does deletes, hardlinks and a final fix of the directory mtimes. In 2.5.5 this output should be disabled, see line 290 of generator.c: /* f_out is set to -1 when doing final directory permission and modification time repair */ if (set_perms(fname,file,NULL,0) && verbose && (f_out != -1)) rprintf(FINFO,"%s/\n",fname); return; Are you running 2.5.5? Craig
Re: rsync feature suggestion
Might it be possible to take the file list that you want to feed to
rsync and turn it into an rsync.conf file?
A simple bash script could create the config file and call rsync (with
the --config= to specify the temporary config file)
Something like this (syntax most likely is wrong, haven't tested it):
#!/bin/sh
IFS="
"
cat /etc/rsync.conf > rsync_command
FILES_TO_SYNC=`cat file_list.txt`
for EACH_FILE in $FILES_TO_SYNC; do
echo ' --include="${EACH_FILE}"' >> rsync_command
done
rsync --config=rsync_command
- Ed King
Hadmut Danisch wrote:
Hi,
I'd like to suggest a new feature to rsync.
Problem:
Currently, rsync generates a recursive list of file
existing a the source directory, modifies this list by
includes and excludes, and then copies these files.
That's pretty good in most, but not all cases.
I am mirroring a debian archive, but unfortunately,
debian mixes all files of several distributions in a
subtree /pool. There is no way to select only the files
of a certain distribution through a simple exclude/include
expression.
There is a tool called debmirror, which first downloads
the distribution index files, extracts all the filenames/paths
of the files needed and then calls rsync for every single file.
Thats certainly not useful, especially since rsync shows the
servers motd for every single file.
Therefore, I'd like to suggest a new option: Allow rsync to
not build the list of files existing at the source directory
by recursively walking through the source directory, but by
reading a file or stdin to get a list of files to be copied.
This would allow to mirror the distribution index files in a
first step, then build the list of files needed and then to
download these files is a second step.
An alternative method would be to keep the recursive method, but
to open a pipe to an external program. Before downloading a
file, the path is printed to the pipe and an answer is read
from the pipe. Thus, an external filter program can decide for
each single file whether to copy it or not.
regards
Hadmut
(Please respond directly, I'm not on your mailing list)
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Rsync --daemon v 2.5.5 v26 causing kernel panic
Has anyone seen this? Looking for past experiences / ideas. Will post progress. I'm tracking down a problem that seems to be caused by rsync. When moving files from a remote server I get a kernel panic. We have a number of servers that back up to a main box -- the panic only occurs when a specific client backs up. It occurs on the box it is backing up to -- not the client. The kernel is Linux 2.4.20 (just compiled -- no patches), files are stored on a 4 disk raid (80GB Western Digital drives, software raid) with Reiserfs. This is being done over a vpn connection controlled by another machine (gateway machine) running tinc so we're not using ssh or any other shell on the rsync machine. I have: recompiled rsync at both locations recompiled the kernel from new source code (panic in 2.4.19, system rebooted in 2.4.20) I will: try different hardware run a file system check at the main system (the one that crashes) exclude directories in the backup (rsync one directory at a time -- see where it crashes) I did notice filenames on the client machine that contain control characters -- but they seem to have backed up before. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Rsync --daemon v 2.5.5 v26 causing kernel panic
After switching much hardware (and getting some helpful suggestions) I moved the specific machine's files on the backup server to a hard drive outside the raid (still on the backup server, /dev/hdi1) and tried rsync -- problem solved. It seems there's a problem with the journaled filesystems (reiserfs) and raid -- reading the files is ok (cp command worked fine), writing them is not. I'm going to turn on reiser debugging and internal checks and re-run -- maybe I can send something to the reiser or kernel people of interest (don't want to go back to ext2) jw schultz wrote: On Fri, Jan 10, 2003 at 02:25:57PM -0600, Edward King wrote: Has anyone seen this? Looking for past experiences / ideas. Will post progress. I'm tracking down a problem that seems to be caused by rsync. When moving files from a remote server I get a kernel panic. We have a number of servers that back up to a main box -- the panic only occurs when a specific client backs up. It occurs on the box it is backing up to -- not the client. The kernel is Linux 2.4.20 (just compiled -- no patches), files are stored on a 4 disk raid (80GB Western Digital drives, software raid) with Reiserfs. This is being done over a vpn connection controlled by another machine (gateway machine) running tinc so we're not using ssh or any other shell on the rsync machine. I have: recompiled rsync at both locations recompiled the kernel from new source code (panic in 2.4.19, system rebooted in 2.4.20) I will: try different hardware run a file system check at the main system (the one that crashes) exclude directories in the backup (rsync one directory at a time -- see where it crashes) I did notice filenames on the client machine that contain control characters -- but they seem to have backed up before. Just to confirm: You are doing backup-server initiated pull something like "rsync server::module destdir" and the machine you execute this on (the receiver) panics. Rsync will of course not be the culprit. However, rsync is very good at stressing a system and the kernel developers have found it to be a common test case. Most likely it is a hardware fault. Probably timing sensitive. I'd check the logs for oopses and and disk errors. Also try downgrading the mode with hdparm. If this isn't a obvious hardware fault you should report it to the linux-kernel people. Look on www.kernelnewbies.org for instructions. This will be of interest to the developers of md, reiserfs and the specific IDE driver.
Re: Rsync --daemon v 2.5.5 v26 causing kernel panic
Just did a successful rsync -- problem was with reiser filesystem. Moved all files off the raid, ran reiserfsck with --rebuild-tree (fix-fixable was not enough), ran rsync after rebuild finished -- no problems. There were a number of files that the rebuild-tree found that weren't attached Edward King wrote: After switching much hardware (and getting some helpful suggestions) I moved the specific machine's files on the backup server to a hard drive outside the raid (still on the backup server, /dev/hdi1) and tried rsync -- problem solved. It seems there's a problem with the journaled filesystems (reiserfs) and raid -- reading the files is ok (cp command worked fine), writing them is not. I'm going to turn on reiser debugging and internal checks and re-run -- maybe I can send something to the reiser or kernel people of interest (don't want to go back to ext2) jw schultz wrote: On Fri, Jan 10, 2003 at 02:25:57PM -0600, Edward King wrote: Has anyone seen this? Looking for past experiences / ideas. Will post progress. I'm tracking down a problem that seems to be caused by rsync. When moving files from a remote server I get a kernel panic. We have a number of servers that back up to a main box -- the panic only occurs when a specific client backs up. It occurs on the box it is backing up to -- not the client. The kernel is Linux 2.4.20 (just compiled -- no patches), files are stored on a 4 disk raid (80GB Western Digital drives, software raid) with Reiserfs. This is being done over a vpn connection controlled by another machine (gateway machine) running tinc so we're not using ssh or any other shell on the rsync machine. I have: recompiled rsync at both locations recompiled the kernel from new source code (panic in 2.4.19, system rebooted in 2.4.20) I will: try different hardware run a file system check at the main system (the one that crashes) exclude directories in the backup (rsync one directory at a time -- see where it crashes) I did notice filenames on the client machine that contain control characters -- but they seem to have backed up before. Just to confirm: You are doing backup-server initiated pull something like "rsync server::module destdir" and the machine you execute this on (the receiver) panics. Rsync will of course not be the culprit. However, rsync is very good at stressing a system and the kernel developers have found it to be a common test case. Most likely it is a hardware fault. Probably timing sensitive. I'd check the logs for oopses and and disk errors. Also try downgrading the mode with hdparm. If this isn't a obvious hardware fault you should report it to the linux-kernel people. Look on www.kernelnewbies.org for instructions. This will be of interest to the developers of md, reiserfs and the specific IDE driver.
log errors using batch reads and writes
I get the following error when "rsync --read-batch" but all the files seem to be updated (spot checking): etc/stats/FileSize is uptodate etc/stats/rsync.log file_flist_entry (819) != flist_entry (842) rsync error: syntax or usage error (code 1) at batch.c(480) rsync: writefd_unbuffered failed to write 65 bytes: phase "unknown": Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(515) Anyone who can shed light on the cause, or at least a way of testing if the sync was successful. Thanks! Details: The last file listed -- rsync.log -- is the log file created when writing the batch. The write command is: rsync -r -a --partial --bwlimit=20 \ --suffix=".backup" --backup --delete \ --write-batch="/smb/Backup/batch" \ --exclude="*.backup" \ --exclude="/proc/" \ --exclude="/var/" \ --exclude="/dev/" \ --exclude="/home/" \ --exclude="/tmp/" \ --exclude="/smb/Backup/" \ --stats -v /. 192.168.1.100::BACKUP/ > rsync.log 2>&1 I have noticed that the batch.rsync_argvs is limited to 255 characters. I restore with the following command: rsync -r -a --partial \ --backup --delete \ --read-batch="batch" \ --exclude="*.backup" \ --stats -vv ../backup/ > rsync.log 2>&1 Edward King -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Error with path names greater than 255 characters
Rsync 2.5.6 has a problem with pathnames or filenames longer than 255. I redefined them in rsync.h, which generated errors on make pointing to the previous definitions which look fine. Any ideas on where to check next? Here's the error I got trying to rsync: receiving file list ... done media/zip.backup.2003-06-03.1415/windows.backup.2003-06-03.1415/oracle.backup.2003-06-03.1415/Dev6i.backup.2003-06-03.1415/win32.backup.2003-06-03.1415/d2dh.backup.2003-06-03.1415/6_0_8_11_4.backup.2003-06-03.1415/doc.backup.2003-06-03.1415/ media/zip.backup.2003-06-03.1415/windows.backup.2003-06-03.1415/oracle.backup.2003-06-03.1415/Dev6i.backup.2003-06-03.1415/win32.backup.2003-06-03.1415/d2dh.backup.2003-06-03.1415/6_0_8_11_4.backup.2003-06-03.1415/doc.backup.2003-06-03.1415/d2ky2kwp4.pdf.backup.2003-06-03.1415 write failed on media/zip.backup.2003-06-03.1415/windows.backup.2003-06-03.1415/oracle.backup.2003-06-03.1415/Dev6i.backup.2003-06-03.1415/win32.backup.2003-06-03.1415/d2dh.backup.2003-06-03.1415/6_0_8_11_4.backup.2003-06-03.1415/doc.backup.2003-06-03.1415/d2ky2kwp4.pdf.backup.2003-06-03.1415 : Success rsync error: error in file IO (code 11) at receiver.c(243) rsync: connection unexpectedly closed (5844537 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) Here's what make produced after redefining the constants: [EMAIL PROTECTED] rsync-2.5.6]# make gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c rsync.c -o rsync.o In file included from rsync.c:23: rsync.h:325: warning: `PATH_MAX' redefined /usr/include/linux/limits.h:14: warning: this is the location of the previous definition rsync.h:326: warning: `MAXPATHLEN' redefined /usr/include/sys/param.h:35: warning: this is the location of the previous definition And checking for the previous values of PATH_MAX and MAXPATHLEN I found: ***/usr/include/linux/limits.h*** #define NAME_MAX 255/* # chars in a file name */ #define PATH_MAX4095/* # chars in a path name */ ***/usr/include/sys/param.h*** #define MAXPATHLEN PATH_MAX Thanks! Ed King -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync error code 12
rsync: writefd_unbuffered failed to write 1 bytes: phase "unknown": Connection timed out rsync error: error in rsync protocol data stream (code 12) at io.c(515) Any idea regarding this error? We sync a number of machines, and one consistently has this error every night. If I exclude the file that the error occurs on, it will go past and fail on another file. One large file it failed on over and over. The connection is over a vpn, but we always can get through and are using bwlimit to throttle down to 15k. Disk space is good on both sides, running as root (uid 0 on the server side), versions 2.5.6 protocol 26 on both sides. Rsync will hang, pings still go through and we can still ssh into the sending server. Both servers are running Linux. Thanks for any suggestions! Ed -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync error code 12 at phase "unknown"
Any suggestions on tracking this problem down? We've had it happen on a few machines -- some through VPN's, one from a Mac OS X to a Linux machine on the local network. Some machines never have this problem. We think it's a hiccup in the network connection -- but it's very minor. We've setup pings and tracked the data sent through the interface and have never seen it drop below 300 bytes / 5 second slice (so we know the connection isn't crashing down on us -- besides, the retry should catch it.) I have checked the usual culprits and think my next step is to wade into the code (disk space good, timeouts virtually nonexistent, cpu load low, lots of free memory, etc.) The phase shows unknown, and the call in io.c is simply a write function -- any suggestions on back tracking to find what is calling the write? Maybe we can restart the phase that's failing (kind of like transaction framing for sql -- perform a commit/rollback operation.) It works great, just seems vulnerable to a lost packet somewhere. Log from "rsync --partial -v -a -z --delete --backup --suffix='.backup' --bwlimit=15 /. 10.1.1.61::/save/": building file list ... done etc/cups/certs/0 etc/mail/statistics etc/mail/logs/092603 rsync: writefd_unbuffered failed to write 16385 bytes: phase "unknown": Connection timed out rsync error: error in rsync protocol data stream (code 12) at io.c(515) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
