Wow can't tell you how many times I've read over the man page but Batch Mode never jumped out at me before.
Batch Mode could very well provide a solution, I'm going to give it a run. Also, I think I found the knob in Ubuntu that's creating the error in rsync. It's /proc/sys/net/ipv4/tcp_retries2 which governs the number of times to attempt to re-transmit a TCP packet. The default value is 15 which provided about a 16 minute window to failure. Turning it down to 5 reduced the window to 20 seconds, where turning it up a little to 8 increased the window to 2 minutes, so it's definitely on a curve. I think with a combination of tuning tcp_retries2 and rsync Batch Mode I may be able to pull together a solution. Regards, Donald On Fri, Jul 15, 2011 at 4:26 PM, Brian K. White <br...@aljex.com> wrote: > On 7/15/2011 2:42 PM, Matthias Schniedermeyer wrote: > >> On 15.07.2011 13:10, Donald Pearson wrote: >> >>> >>> Matthias, >>> >>> A vpn tunnel is an interesting idea. Do you know how long you're able to >>> keep rsync in limbo before it will give up? >>> >> >> I haven't really tried. But it was about 15 Minutes the one time it >> didn't reconnect in time. >> >> The issue I think is keeping the sockets open, thereby keeping the >>> processes >>> active. >>> >> >> The problem is that TCP really isn't made for transfer over an >> unreliable medium. >> >> Besides rsync, the OSes on both sides also have to keep the connction >> running, but i really don't know which knobs have to be turned to extent >> the time after which the OS times out the connect. >> >> My first guess was the tcp_fin_timeout setting of the Ubuntu operating >>> system, but this is set for 60 seconds. Leaving everything alone I see >>> roughly 6 minutes before the rsync client errors out. >>> >> >> tcp_fin is, AFAIK, after you close the socket normaly. >> >> Thanks again for everybody's help and by all means keep the ideas coming. >>> I >>> am trying new angles as I come up with them, I haven't given up on this >>> yet. >>> >> >> I got another idea. >> >> If you have the storage on the source-side, to keep the state of the >> target-side. You can create batch-files (--write-batch& Co.). Of couse >> you also need the storage on the target-side for the batch-files and the >> need space for eventuell growth and temporary files. >> >> You can then copy these batch-files with --append, make sure the >> target-rsync is dead before you retry, for an added bonus i would MD5 >> the file on the source side and veryify it after copying. After you have >> transferred a complete batch-file, you can apply it. (Use 'screen' if >> you do it remotely) >> > > That's about what I was trying to suggest myself. Let rsync generate delta > data locally-only with the batch option, then let bittorrent deliver the > delta data over the crappy network, then let rsync apply the delta locally > on the other end. > > -- > bkw > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/** > mailman/listinfo/rsync <https://lists.samba.org/mailman/listinfo/rsync> > Before posting, read: > http://www.catb.org/~esr/faqs/**smart-questions.html<http://www.catb.org/~esr/faqs/smart-questions.html> >
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html