On 7/12/2011 11:10 AM, Donald Pearson wrote:
@Eberhard: I understand what you're trying to say, but in this
environment the reality is rsync reaches and impasse where it is unable
to get beyond work that has already been completed before link failure
cuts it off again.
@Leen: A combination of --append with --partial is what I tried,
however --partial appears to be ignored with the --append switch is
given. A partial file is not generated.
--inplace leaves me with the same problem. Rsync will still start from
the beginning every time, which is not a behavior this network
environment can support. --append unfortunately implies --inplace,
which I'm sure is the reason it is incompatible with --partial.
@Matthias: --timeout actually causes more problems than it solves.
Without --timeout set, the server (target-side) rsync process
continues to run, apparently indefinitely, waiting for the client
(source-side) to talk to it again. This behavior would be excellent,
except that it seems that it will only talk to the original rsync
client. If the original rsync client process ends, and a new one is
started, the original rsync server process does not respond to it, a new
rsync server process is created to speak with the new rsync client
process, and the whole transaction is unique from the original
transaction in every way. This means if --partial is an included
switch, you will have 2 partial files created. The original partial
file will remain a partial file indefinitely. The 2nd process will not
pick up where the 1st process ended, it simply starts all over again
from the beginning.
When --timeout is set, both the server and client processes will wait
the specified number of seconds, and then give up. When --partial is
used, the server will over-write the destination file with the partial
file. This is absolutely detrimental to sending only the differences.
Here is a file list of the server side directory during and after the
command 'rsync -hvzB=512 --partial --progress --timeout=30
/images/test/AllCafeFinal_2011-06-28.temp
192.168.0.27:/images/test/AllCafeFinal_2011-06-28.temp
A few moments after rsync has begun, the network cable between the two
hosts is removed.
Before:
analyst@ubuntudesktop1010:/images/test$ ls -lah
total 15G
drwxr-xr-x 2 analyst analyst 64K 2011-07-12 10:35 .
drwxr-xr-x 3 analyst root 48K 2011-07-07 16:05 ..
-rw-r--r-- 1 analyst analyst 6.1G 2011-07-12 10:13
AllCafeFinal_2011-06-28.temp
-rw-r--r-- 1 analyst analyst 3.3G 2011-07-11 14:00
AllCafeFinal_2011-06-28.temp.gz
-rw-r--r-- 1 analyst analyst 3.3G 2011-04-01 13:59
AllCafeFinal_2011-06-28.temp.gz.bak
-rw------- 1 analyst analyst 582M 2011-07-12 10:36
.AllCafeFinal_2011-06-28.temp.S6fwyP
-rw-r--r-- 1 root root 1.1G 2011-07-11 11:23 rsync.pcap
As you can see the destination file is 6.1 Gigs in size. The partial
file has grown to 582 Megabytes.
Now the network cable is pulled, and after 30 seconds both rsync client
and servers time out. By the way the source file is 7.5 Gigs in this
scenario.
After:
analyst@ubuntudesktop1010:/images/test$ ls -lah
total 8.3G
drwxr-xr-x 2 analyst analyst 64K 2011-07-12 10:37 .
drwxr-xr-x 3 analyst root 48K 2011-07-07 16:05 ..
-rw-r--r-- 1 analyst analyst 583M 2011-07-12 10:37
AllCafeFinal_2011-06-28.temp
-rw-r--r-- 1 analyst analyst 3.3G 2011-07-11 14:00
AllCafeFinal_2011-06-28.temp.gz
-rw-r--r-- 1 analyst analyst 3.3G 2011-04-01 13:59
AllCafeFinal_2011-06-28.temp.gz.bak
-rw-r--r-- 1 root root 1.1G 2011-07-11 11:23 rsync.pcap
analyst@ubuntudesktop1010:/images/test$
Now you can see the original 6.1 Gig destination file is gone. It has
been replaced with The 583 Megabyte partial file. If I run rsync again,
rsync is going to have to send the entire 7 gigs remainder. It can no
longer only send the difference between the source and destination,
because the destination has been destroyed.
--inplace leaves me with the issue of needing to re-check data that has
already been sent, every time. I need to avoid that. --append implies
--inplace and so inherits it's issues so that also isn't an option.
As far as I can tell, rsync is unable to resume a failed transfer. At
best rsync starts a fresh transfer over again, but by design doesn't
need to re-send what it already sent. Of course I am very much open to
being wrong about this statement and I appreciate everyone who has taken
the time to consider my issue and offer suggestions.
3 potential ways for a true resume to be accomplished come to mind.
- keep the indefinite timeout of rsync server, but enable it to
respond to new client connections so it can continue to write to the
same partial file
- add a feature that allows rsync to memory dump on timeout, so that a
new rsync process can use that memory dump data to truly pick up where
the previous process stopped.
- modify --append such that --append no longer implies --inplace, in a
manner that is also compatible with --partial. Right now the observed
behavior of --append is that the destination file size is looked at.
Rsync then skips that much data from the front of the source file. If
--append is given with --partial, the existence of a partial file will
want to be verified, and then the size of that partial file will want to
be looked at, not the destination file. The rsync client can then skip
that much data (as much as already exists in the partial file) of the
source file, and append to the partial file, not the destination file
directly. This also means the partial file cannot over-write the
destination file until it is a true copy of the source.
Unfortunately I am incapable of enabling any of these capabilities myself.
Regards,
Donald
On Tue, Jul 12, 2011 at 3:52 AM, Matthias Schniedermeyer <m...@citd.de
<mailto:m...@citd.de>> wrote:
On 11.07.2011 16:01, Donald Pearson wrote:
> I am looking to do state-full resume of rsync transfers.
>
> My network environment is is an unreliable and slow satellite
> infrastructure, and the files I need to send are approaching 10
gigs in
> size. In this network environment often times links cannot be
maintained
> for more than a few minutes at a time. In this environment,
bandwidth is at
> a premium, which is why rsync was chosen as ideal for the job.
>
> The problem that I am encountering while using rsync in these
conditions is
> that the connection between the client and server will drop due
to network
> instability before rsync can transfer the entire file.
>
> Upon retries, rsync starts from the beginning. Re-checking data
that has
> already been sent, as well as re-building the checksum in it's
entirety
> every time. Eventually I reach an impasse where the frequency of
link loss
> prevents rsync from ever getting any new data to the destination.
>
> I've been reading through the various switches in the man pages
to try to
> find a combination that will work. My thinking was to use a
combination of
> --partial and --append. With the first attempt using the
--partial switch,
> and subsequent attempts using both --partial and --append. The
idea being
> rsync would build a new "partial" file, and be able to resume
building that
> file while making the assumption upon subsequent retries that the
existing
> partial file, however large it may be, was assembled correctly
and does not
> need to be checked.
>
> However in practice rsync does not work in this way.
I think you didn't wait for the target rsync to complete, if a
connection breaks, you have 2 parts left hanging. The less visible
target-side is the important one here. That rsync has to "complete"
before you do another try. Depending on how your connection drops it MAY
hang for some time. I don't remember if rsync does "the right thing"
if you just kill it, or if you have to wait for it. In the latter case
"--timeout" sounds like it can be used to expedite matters.
And also --inplace, with or without --append, reads like it is what you
want, if you can live with it's caveats.
bittorrent may be a kludgy way around if it turns out to be impossible
to get rsync to handle this situation. You don't care about the
distributed aspect of course, but it builds files in tiny random chunks
as they become available and suffers unplanned loss of connection and
restarts and final checksum before presenting the file as 100% done.
Which is all exactly what you need. I don't know if bt does any
rsync-like work to avoid sending the same data over when a source file
is updated. So each transfer will take the full time, but each transfer
will chip away until it's complete no matter how bad the network.
For that matter, so will http/ftp resume and wget/curl/ncftp etc..
You may also simply break up the file into smaller pieces (using cut or
dd in a script) into a temp dir rsync that directory in a loop until it
succeeds, and alter the script to use smaller chunks until it succeeds
as fast as it's going to. If you're dealing with more than a few files
then I admit it get's pretty involved writing a script to collect a file
list and do that for each file, reliably.
Or you may be able to use rsync batch to generate the delta data but not
transfer it anywhere, and use bt or http/ftp with resume to transfer the
delta data. I've never used batch so you can read the same manual I'd
have to to actually try that idea.
--
bkw
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
