Rsync stops randomly (using vanished files bash script)
Hey, I've been trying to transfer about 90GB from one Server to another using rsync. But rsync stops randomly. I use the bash script provided in a bug report, to block errors about vanished files! Here is how I start rsync: ./rsync-no-vanished -v -a -e ssh --delete --progress -z [EMAIL PROTECTED]:/var/www/users /data/oldkilu3/ > /data/rsync.kilu3n & When I log in into the server after a couple of hours, rsync is not running anymore and has transfered only like 10% of the files. I tried it 5 times over 3 days and both servers were definitely up and running the whole time. Here are some the tails of 3 of my output files: # tail -f /data/rsync.kilu2n2 data recv 16378 at 78016976 data recv 16378 at 78033354 data recv 52 at 78049732 data recv 16378 at 78049784 data recv 16378 at 78066162 data recv 45 at 78082540 data recv 16378 at 78082585 data recv 16378 at 78098963 data recv 43 at 78115341 data rec # tail -f /data/rsync.kilu3n recv_file_name(users/heinicke) [sender] make_file(users/zhuzhenhua77,*,2) recv_file_name(users/zhuzhenhua77) [sender] make_file(users/langhoff,*,2) recv_file_name(users/langhoff) [sender] make_file(users/neal,*,2) recv_file_name(users/neal) [sender] make_file(users/qingdu,*,2) recv_file_name(users/qingdu) [sender] make_file(users/fuch # tail -f /data/rsync.kilu2n [sender] make_file(users/jugendkreis/forum/images/smiles/icon_arrow.gif,*,2) recv_file_name(users/jugendkreis/forum/images/smiles/icon_arrow.gif) [sender] make_file(users/jugendkreis/forum/images/smiles/icon_biggrin.gif,*,2) recv_file_name(users/jugendkreis/forum/images/smiles/icon_biggrin.gif) [sender] make_file(users/jugendkreis/forum/images/smiles/icon_confused.gif,*,2) recv_file_name(users/jugendkreis/forum/images/smiles/icon_confused.gif) [sender] make_file(users/jugendkreis/forum/images/smiles/icon_cool.gif,*,2) recv_file_name(users/jugendkreis/forum/images/smiles/icon_cool.gif) [sender] make_file(users/jugendkreis/forum/images/smiles/icon_cry.gif,*,2) recv_file_name(users The files are just cut of, and I don't see why. Could anybody explain this to me? Regards, Samy -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync stops randomly (using vanished files bash script)
I don't think this is possible. Pretty much no services except ssh are running on the machine that I'm running the rsync client on. The partition I'm transferring the files to hast 400GB of space left and 3.7 GB of my 4GB memory are still free... Any other suggestions? Do you think it could be possible that the machine which I'm fetching the data from run out of ram? I think the the client should have thrown some error message at least. Regards, Samy Andre Majorel schrieb: On 2007-08-23 18:38 +0200, Samuel Vogel wrote: When I log in into the server after a couple of hours, rsync is not running anymore and has transfered only like 10% of the files. I tried it 5 times over 3 days and both servers were definitely up and running the whole time. Ran out of memory ? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync stops randomly (using vanished files bash script)
At 06:58 24.08.2007 +0200, Samuel Vogel wrote: >I don't think this is possible. Pretty much no services except ssh are running >on the machine that I'm running the rsync client on. The partition I'm >transferring the files to hast 400GB of space left and 3.7 GB of my 4GB memory >are still free... >Any other suggestions? > >Do you think it could be possible that the machine which I'm fetching the data >from run out of ram? I think the the client should have thrown some error >message at least. What rsync version are you using? Up to he last official (2.6.9) there seems to be an error that rsync can deadlock, see here: http://marc.info/?l=rsync&m=116932447411768&w=2 I also had this problem, especially with big files over ssh. I now tried the current cvs version and it works much better, it never stalled anymore in the last few days. bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync stops randomly (using vanished files bash script)
Thanks for the reply. Do you mean on the rsync server or client side? The client has 4GB of Ram, the server only has 1GB. I try to download from the server to the client. I'm sure that there is no file bigger than 500MB. And there can't be any files 1GB, because we have 1GB quotas in place! Like mentioned in the other reply to my question. I will try to use the latest CVS snapshot! I'll report if I have success... Andre Majorel schrieb: On 2007-08-24 06:58 +0200, Samuel Vogel wrote: I don't think this is possible. Pretty much no services except ssh are running on the machine that I'm running the rsync client on. The partition I'm transferring the files to hast 400GB of space left and 3.7 GB of my 4GB memory are still free... Any other suggestions? Do you think it could be possible that the machine which I'm fetching the data from run out of ram? I think the the client should have thrown some error message at least. If you have directories with many files, rsync becomes a memory hog. I've had it bring down a 1-GB machine while copying a 10-GB news spool. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync stops randomly (using vanished files bash script)
On 2007-08-24 12:02 +0200, Samuel Vogel wrote: > Andre Majorel schrieb: > > >If you have directories with many files, rsync becomes a memory > >hog. I've had it bring down a 1-GB machine while copying a 10-GB > >news spool. > > Thanks for the reply. Do you mean on the rsync server or client > side? I don't recall from which machine the transfer was initiated but I think the side that hogs memory is the one where the large directory exists. So the first time you run rsync, it's just on the source host. On subsequent runs, it's on both hosts. > I'm sure that there is no file bigger than 500MB. And there > can't be any files 1GB, because we have 1GB quotas in place! IME, rsync has no problems with large files or even many files. What it can't deal with is single directories with many files. The algorithm used must require holding in memory some per-file data for *all* files in the directory being synced. If you have a news spool, a big maildir folder or some such monstrosity, you might want to try excluding them from the sync and see if it makes a difference. Good luck. -- André Majorel http://www.teaser.fr/~amajorel/> Do not use this account for regular correspondence. See the URL above for contact information. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync stops randomly (using vanished files bash script)
At 16:43 24.08.2007 +0200, Samuel Vogel wrote: >I installed the latest CVS Version (today about 1pm). Again, the same problem >occured... Did you have it on both sides? Did rsync say "sending incremental file list" (probably only with -v)? It only does so if the rsync on the other side understands it as well. bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync stops randomly (using vanished files bash script)
I installed the latest CVS Version (today about 1pm). Again, the same problem occured... Andre Majorel schrieb: On 2007-08-24 12:02 +0200, Samuel Vogel wrote: Andre Majorel schrieb: If you have directories with many files, rsync becomes a memory hog. I've had it bring down a 1-GB machine while copying a 10-GB news spool. Thanks for the reply. Do you mean on the rsync server or client side? I don't recall from which machine the transfer was initiated but I think the side that hogs memory is the one where the large directory exists. So the first time you run rsync, it's just on the source host. On subsequent runs, it's on both hosts. I'm sure that there is no file bigger than 500MB. And there can't be any files 1GB, because we have 1GB quotas in place! IME, rsync has no problems with large files or even many files. What it can't deal with is single directories with many files. The algorithm used must require holding in memory some per-file data for *all* files in the directory being synced. If you have a news spool, a big maildir folder or some such monstrosity, you might want to try excluding them from the sync and see if it makes a difference. Good luck. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync stops randomly (using vanished files bash script)
I installed the CVS version on both sides! Fabian Cenedese schrieb: At 16:43 24.08.2007 +0200, Samuel Vogel wrote: I installed the latest CVS Version (today about 1pm). Again, the same problem occured... Did you have it on both sides? Did rsync say "sending incremental file list" (probably only with -v)? It only does so if the rsync on the other side understands it as well. bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync stops randomly (using vanished files bash script)
On 8/23/07, Samuel Vogel <[EMAIL PROTECTED]> wrote: > I've been trying to transfer about 90GB from one Server to another using > rsync. But rsync stops randomly. I use the bash script provided in a bug > report, to block errors about vanished files! I wonder if this is the same problem that caused rsync to quit with the message about exit code 20. Is the disk almost full, or is there a quota or file-size resource limit that might be preventing rsync from writing to the log? Does rsync work correctly if you call it directly without the rsync-no-vanished script? Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
