Re: Rsync stops inmid of synchronisation
Am 13.03.2012 21:00, schrieb David Sastre Medina: On Mon, Mar 12, 2012 at 01:48:45PM +0100, Richard Ivarson wrote: Is there a way to increase the verbosity level of the sending rsync (aside the parameter -v which we can increase to -vv or even -vvv) ? Or could rsync print further helpful information about what it is doing right now ? Because that could help me. With -vvv rsync prints information about the files it's sending to the remote rsync, but in my case when it has rsynced for a while it suddenly stops to print and do anything, and I got no idea what is happening... Maybe overkill, but you could try to strace the process. Thank you David for this hint. I've never used strace before but I'll look into it and try to use it for my rsync problems. In case I find out some new information I'm going to post them here. -Richard -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Rsync stops inmid of synchronisation
OK, so I called strace -p ID where ID is the pid of a running rsync command. Then I waited until the rsync command timed out due to the unknown problem which I try to encircle. In short the problem is: two computers with WinXP_pro 32bit and up-to-date Cygwin are connected, and I try to rsync some folders of the first computer to the remote computer. However, after the sender rsynced some thousands of smaller files correctly (respectivly: it checked them without transfering the up-to-date files), it suddently stops to do anything... That's why I've added the --timeout=30 parameter to the sending rsync, so that my rsync aborts the offending folders and the bash script can continue with the following rsync commands. So now rsync said this: [sender] io timeout after 30 seconds -- exiting rsync error: timeout in data send/receive (code 30) at /home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/io.c(140) [sender=3.0.9] And the strace log which happened before and during the time-out is attached below. (I cut the middle part of it because it just repeated about 250 times the _set_errno fstat64 lines.) Unfortunately the log doesn't say anything to me, but maybe some reader knowing the internas of Cygwin and rsync command in particular, sees more? Thank you very much! -Richard Am 13.03.2012 21:00, schrieb David Sastre Medina: On Mon, Mar 12, 2012 at 01:48:45PM +0100, Richard Ivarson wrote: Is there a way to increase the verbosity level of the sending rsync (aside the parameter -v which we can increase to -vv or even -vvv) ? Or could rsync print further helpful information about what it is doing right now ? Because that could help me. With -vvv rsync prints information about the files it's sending to the remote rsync, but in my case when it has rsynced for a while it suddenly stops to print and do anything, and I got no idea what is happening... Maybe overkill, but you could try to strace the process. [..] 22 569009835 [main] rsync 2200 cygwin_select: sel.always_ready 0 36 569009871 [main] rsync 2200 start_thread_socket: stuff_start 0x228208 36 569009907 [socksel] rsync 2200 thread_socket: stuff_start 0x228208 32 569009939 [socksel] rsync 2200 peek_socket: read_ready: 0, write_ready: 0, except_ready: 0 39 569009978 [main] rsync 2200 select_stuff::wait: m 3, ms 15000 14999596 584009574 [main] rsync 2200 select_stuff::cleanup: calling cleanup routines 143 584009717 [main] rsync 2200 socket_cleanup: si 0x2004F1C8 si-thread 0x61186390 78 584009795 [socksel] rsync 2200 thread_socket: leaving thread_socket 108 584009903 [main] rsync 2200 socket_cleanup: returning 56 584009959 [main] rsync 2200 select_stuff::wait: timed out 55 584010014 [main] rsync 2200 select_stuff::wait: returning 1 55 584010069 [main] rsync 2200 cygwin_select: 0 = select(4, 0x2282D8, 0x2282D0, 0x228180, 0x2282C8) 61 584010130 [main] rsync 2200 select_stuff::cleanup: calling cleanup routines 56 584010186 [main] rsync 2200 select_stuff::destroy: deleting select records 79 584010265 [main] rsync 2200 time: 1331803941 = time(0) 185 584010450 [main] rsync 2200 fhandler_console::write: 2265F0, 47 60 584010510 [main] rsync 2200 fhandler_console::write: at 91([) state is 0 87 584010597 [main] rsync 2200 fhandler_console::write: 47 = fhandler_console::write(...) 58 584010655 [main] rsync 2200 write: 47 = write(2, 0x2265F0, 47) 224 584010879 [main] rsync 2200 fhandler_console::write: 61183757, 1 55 584010934 [main] rsync 2200 fhandler_console::write: at 10(0x20) state is 0 169 584011103 [main] rsync 2200 fhandler_console::write: 1 = fhandler_console::write(...) 62 584011165 [main] rsync 2200 write: 1 = write(2, 0x61183757, 1) 248 584011413 [main] rsync 2200 sigaction_worker: signal 30, newact 0x4641DC (handler 0x1), oa 0x419660 67 584011480 [main] rsync 2200 sig_send: sendsig 0x77C, pid 2200, signal -30, its_me 1 60 584011540 [main] rsync 2200 sig_send: wakeup 0x6B8 64 584011604 [main] rsync 2200 sig_send: Waiting for pack.wakeup 0x6B8 85 584011689 [sig] rsync 2200 wait_sig: signalling pack.wakeup 0x6B8 64 584011753 [main] rsync 2200 sig_send: returning 0x0 from sending signal -30 59 584011812 [main] rsync 2200 sigaction_worker: 0 = sigaction(30, 0x4641DC, 0x0) 58 584011870 [main] rsync 2200 sigaction_worker: signal 31, newact 0x4641DC (handler 0x1), oa 0x419FF0 65 584011935 [main] rsync 2200 sig_send: sendsig 0x77C, pid 2200, signal -31, its_me 1 65 584012000 [main] rsync 2200 sig_send: wakeup 0x6B8 60 584012060 [main] rsync 2200 sig_send: Waiting for pack.wakeup 0x6B8 3 584012063 [sig] rsync 2200 wait_sig: signalling pack.wakeup 0x6B8 108 584012171 [main] rsync 2200 sig_send: returning 0x0 from sending signal -31 58 584012229 [main] rsync 2200 sigaction_worker: 0 = sigaction(31, 0x4641DC, 0x0) 133 584012362 [main] rsync 2200 fhandler_console::write: 226550, 128 56 584012418 [main] rsync 2200
Re: Rsync stops inmid of synchronisation
On Mon, Mar 12, 2012 at 01:48:45PM +0100, Richard Ivarson wrote: Is there a way to increase the verbosity level of the sending rsync (aside the parameter -v which we can increase to -vv or even -vvv) ? Or could rsync print further helpful information about what it is doing right now ? Because that could help me. With -vvv rsync prints information about the files it's sending to the remote rsync, but in my case when it has rsynced for a while it suddenly stops to print and do anything, and I got no idea what is happening... Maybe overkill, but you could try to strace the process. -- Huella de clave primaria: AD8F BDC0 5A2C FD5F A179 60E7 F79B AB04 5299 EC56 signature.asc Description: Digital signature
Re: Rsync stops inmid of synchronisation
Is there a way to increase the verbosity level of the sending rsync (aside the parameter -v which we can increase to -vv or even -vvv) ? Or could rsync print further helpful information about what it is doing right now ? Because that could help me. With -vvv rsync prints information about the files it's sending to the remote rsync, but in my case when it has rsynced for a while it suddenly stops to print and do anything, and I got no idea what is happening... Thanks for any help or hints. -Richard Am 06.03.2012 11:02, schrieb Richard Ivarson: LMH, thank you for your hints. So I checked all the drives and files and they're good, as well as the other hardware. Also the file's permissions are good. The idea to sync just one folder after the other is good. It turned out that those few folders with many smaller files (up to 35000 files, mainly man pages) cause the hang of rsync often, but not always because with a certain chance these folders sync fine. That makes it very hard to find the real error. My syncing script now uses separate rsync commands (i.e. lines in the script) for all the problematic folders in contrast to before where all the folders have been separated by spaces in one rsync command. In combination with rsync's timeout parameter (see below please) only those rsync commands with an offending folder are being skipped when the timeout happens, so that the remaining rsync commands can continue to run. This is an improvement. :-) This is what I found out in detail, so far: - If I rsync the problematic folder to a local device, rsync always runs fine! - If I rsync the same problematic folder to the remote computer (on a local network), the problematic folder makes the sending rsync to hang often. So it can't be a problem with the files, it must be some rsync network stuff causing the hang. - If I add the --dry-run parameter, then interestingly rsync hangs too at about the same position in the problematic folder, i.e. without actually having send content data (just information data?) to the receiver. - So is it the sender or receiver which causes the hang? I think the sender, because when I add the rsync parameter --timeout=30 the rsync command is being terminated/skipped at the offending folder and continues with the next rsync command in the script. When however I use the parameter --contimeout=30 instead, the scripts waits forever -- as it happens with no timeout parameter. So in the end I still don't see what's going on, unfortunately. -Richard LMH wrote: One thing to check is the disk drives. I have had rsync stop when it reaches corrupted sectors, especially if those sectors corrupt part of the file system. I don't remember anything helpful in the log files, I just noticed that it wasn't finishing. Try running disk diagnostic software and make sure your hardware is good. What kind of hard drives do you have and what OS? If that doesn't show up anything, I would start by synching just one folder. Create a test folder on both computers and add a few files. Make sure your permissions are correct. Make a change in one file and then run rsync. If it finishes and the logs look good, add more to the test folder and see if you can find where it brakes. You can test things like very large files, and folders with a large number of files. If you can't get it to fail, then the issue may be with the data on one of the computers. The problem may also be with the rsync software, but I can't advise well on that end. LMH -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Rsync stops inmid of synchronisation
LMH, thank you for your hints. So I checked all the drives and files and they're good, as well as the other hardware. Also the file's permissions are good. The idea to sync just one folder after the other is good. It turned out that those few folders with many smaller files (up to 35000 files, mainly man pages) cause the hang of rsync often, but not always because with a certain chance these folders sync fine. That makes it very hard to find the real error. My syncing script now uses separate rsync commands (i.e. lines in the script) for all the problematic folders in contrast to before where all the folders have been separated by spaces in one rsync command. In combination with rsync's timeout parameter (see below please) only those rsync commands with an offending folder are being skipped when the timeout happens, so that the remaining rsync commands can continue to run. This is an improvement. :-) This is what I found out in detail, so far: - If I rsync the problematic folder to a local device, rsync always runs fine! - If I rsync the same problematic folder to the remote computer (on a local network), the problematic folder makes the sending rsync to hang often. So it can't be a problem with the files, it must be some rsync network stuff causing the hang. - If I add the --dry-run parameter, then interestingly rsync hangs too at about the same position in the problematic folder, i.e. without actually having send content data (just information data?) to the receiver. - So is it the sender or receiver which causes the hang? I think the sender, because when I add the rsync parameter --timeout=30 the rsync command is being terminated/skipped at the offending folder and continues with the next rsync command in the script. When however I use the parameter --contimeout=30 instead, the scripts waits forever -- as it happens with no timeout parameter. So in the end I still don't see what's going on, unfortunately. -Richard LMH wrote: One thing to check is the disk drives. I have had rsync stop when it reaches corrupted sectors, especially if those sectors corrupt part of the file system. I don't remember anything helpful in the log files, I just noticed that it wasn't finishing. Try running disk diagnostic software and make sure your hardware is good. What kind of hard drives do you have and what OS? If that doesn't show up anything, I would start by synching just one folder. Create a test folder on both computers and add a few files. Make sure your permissions are correct. Make a change in one file and then run rsync. If it finishes and the logs look good, add more to the test folder and see if you can find where it brakes. You can test things like very large files, and folders with a large number of files. If you can't get it to fail, then the issue may be with the data on one of the computers. The problem may also be with the rsync software, but I can't advise well on that end. LMH -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Rsync stops inmid of synchronisation
Good day, I've successfully used Cygwin and in particular it's RSYNC for many years, for example to sync a Windows XP computer named Sendi to another XP computer named Desti in a local network. Since a few months however rsync is causing me endless trouble. I regularly upgrade to new versions of Cygwin, so I'm up to date with all Cygwin programs. Now I don't know if my network is the problem (it works fine for all other purposes except rsync) or the rsync program (or its configuration). Maybe somebody knows how I could encircle the problem. Here's the details: - Sendi and Desti are connected via a Ethernet cable to a Router - On Desti rsync runs via rsync --daemon - On Sendi the rsync task is started with rsync foldername/ Desti::modulename/ (and module name is configured as described in Cygwin's or rsync's manual) When the rsync tasks start, at first everything runs fine. I see this by following to Desti's rsync log file and Sendi's verbose output, and at the files being synchronisised. Ie the two rsync programs exchange the files which are out of sync -- but unfortunately just up to a certain point. After a number of files have been transmitted -- and this number varies every time I re-start the task (sometimes it's some 1000 files, sometimes less) -- the two rsync programs stop to talk to each other. Sendi's not continuing to send file names anymore. But why? I've increased Sendi's rsync verbose level to maximum but it just stops to print the file names it wants to send to Desti. So I've started Wireshark to see the network traffic, and next to good RSYNC packages there are also some bad ones named Malformed RSYNC packet in Wireshark. For example Sendi's Wireshark log looks like this: [Malformed Packet: RSYNC] Expert Info (Error/Malformed): Malformed Packet (Exception occurred) Message: Malformed Packet (Exception occurred) Severity level: Error Group: Malformed Since I'm no network expert (I can just start Wireshark and watch the many log entries), does anybody of you Cygwin users know what's happening? To sum up the problem: rsync worked well on Sendi Desti, when suddenly a started rsync task just partly works but then stops. And depending on how many times I re-start the task or the computers, I can get smaller or bigger parts of the whole sync task. Usually not the entire however, because it stops long before the end... Thanks for any hints. -Richard -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Rsync stops inmid of synchronisation
One thing to check is the disk drives. I have had rsync stop when it reaches corrupted sectors, especially if those sectors corrupt part of the file system. I don't remember anything helpful in the log files, I just noticed that it wasn't finishing. Try running disk diagnostic software and make sure your hardware is good. What kind of hard drives do you have and what OS? If that doesn't show up anything, I would start by synching just one folder. Create a test folder on both computers and add a few files. Make sure your permissions are correct. Make a change in one file and then run rsync. If it finishes and the logs look good, add more to the test folder and see if you can find where it brakes. You can test things like very large files, and folders with a large number of files. If you can't get it to fail, then the issue may be with the data on one of the computers. The problem may also be with the rsync software, but I can't advise well on that end. LMH Richard Ivarson wrote: Good day, I've successfully used Cygwin and in particular it's RSYNC for many years, for example to sync a Windows XP computer named Sendi to another XP computer named Desti in a local network. Since a few months however rsync is causing me endless trouble. I regularly upgrade to new versions of Cygwin, so I'm up to date with all Cygwin programs. Now I don't know if my network is the problem (it works fine for all other purposes except rsync) or the rsync program (or its configuration). Maybe somebody knows how I could encircle the problem. Here's the details: - Sendi and Desti are connected via a Ethernet cable to a Router - On Desti rsync runs via rsync --daemon - On Sendi the rsync task is started with rsync foldername/ Desti::modulename/ (and module name is configured as described in Cygwin's or rsync's manual) When the rsync tasks start, at first everything runs fine. I see this by following to Desti's rsync log file and Sendi's verbose output, and at the files being synchronisised. Ie the two rsync programs exchange the files which are out of sync -- but unfortunately just up to a certain point. After a number of files have been transmitted -- and this number varies every time I re-start the task (sometimes it's some 1000 files, sometimes less) -- the two rsync programs stop to talk to each other. Sendi's not continuing to send file names anymore. But why? I've increased Sendi's rsync verbose level to maximum but it just stops to print the file names it wants to send to Desti. So I've started Wireshark to see the network traffic, and next to good RSYNC packages there are also some bad ones named Malformed RSYNC packet in Wireshark. For example Sendi's Wireshark log looks like this: [Malformed Packet: RSYNC] Expert Info (Error/Malformed): Malformed Packet (Exception occurred) Message: Malformed Packet (Exception occurred) Severity level: Error Group: Malformed Since I'm no network expert (I can just start Wireshark and watch the many log entries), does anybody of you Cygwin users know what's happening? To sum up the problem: rsync worked well on Sendi Desti, when suddenly a started rsync task just partly works but then stops. And depending on how many times I re-start the task or the computers, I can get smaller or bigger parts of the whole sync task. Usually not the entire however, because it stops long before the end... Thanks for any hints. -Richard -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
