Re: [Bug 5124] Parallelize the rsync run using multiple threads and/or connections

[Jason Haar]

On 26/01/14 18:03, L.A. Walsh wrote:
> But multiple TCP connections are not used to load a single picture. 
> They are used
> for separate items on the page.  A single TCP stream CAN be very fast
> and rsync
> isn't close to hitting that limit.
> The proof?   Using 1Gb connections, smb/cifs could get 125MB/s writes and
> 119MB/s reads -- the writes were at "theoretical speeds" and were
> faster, because
> the sender doesn't have to wait for the ACKs with a large window size.
>

A bit late but I'll add my 2c worth.

bbcp - multi-tcp/threaded application. Completely nails rsync when
transferring over high-bandwidth/high-latency links

http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html

Like bittorrent, it establishes multiple TCP channels between a bbcp
client and server, and I guess has a parent process that tells each
child what part of the directory structure/data stream it is responsible
for, and joins it all up at the other end

I have tested rsync over a 100Mbs continental link and am lucky to get
10Mbs. Using bbcp with 4-6 channels, I can get 40-50Mbs (that's on a
link with other real traffic on it - so it may have actually got
80-90Mbs byt itself for all I know)

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Re[2]: Rsync performance with large exchange database files

[Jonathan Aquilina]

Bruce there is also bacula which seems to be available for all the os's you
are running.

http://www.bacula.org/en/?page=documentation


On Mon, Feb 10, 2014 at 11:24 PM,  wrote:

>
> Well the local test was just to a test to see if I could understand why
> the remote sync of the exchange database was so slow.  I've heard that
> rsync is less efficient for local copies but this isn't like 80% the
> performance, or half the performance.. It's a massive difference - which I
> wasn't expecting to see.
>
> I have looked at unison but I also backup several linux and bsd systems
> here and rsync just seemed like a good choice since it's still be developed
> and it works on everything.
>
> In the end, I'm still looking to understand why a sync of a large exchange
> database file is taking 30-40 hours to finish and if there's anything I can
> do to help reduce that window.  I can't add a cache drive to the FreeBSD
> server very easily at the moment so I was trying to narrow down if the
> issue is on the BSD side or Windows side or maybe a mix of both.
>
> I wish a native windows client of rsync existed :)
>
> -- Original Message --
> From: "Kevin Korb" 
> To: rsync@lists.samba.org
> Sent: 2/10/2014 4:09:15 PM
> Subject: Re: Rsync performance with large exchange database files
>
>  -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Rsync is known to be pretty inefficient on local copies (-W is forced
>> there btw) and cygwin doesn't really help with that either.
>> Essentially, when not networking rsync isn't much smarter than cp but
>> it has a ton of extra overhead.
>>
>> Also, maybe you want unison since there is a native Windows version of it?
>>
>> On 02/10/2014 05:05 PM, br...@sqls.net wrote:
>>
>>>
>>>  Clean copy. I even used the -W flag to see if it made a difference
>>>  but, nope.
>>>
>>>  I'm testing this same test on some of my other servers too. See if
>>>  there's any common-ground I can find.
>>>
>>>  On another servers (MS SQL Server) with faster disks I tried a
>>>  similar test just now. There's only the C drive on this server but
>>>  I used my same test file from the other server and used rsync
>>>  (3.1.0) to copy the folder from one folder to another folder and it
>>>  kicked off and got up to about 25MB/sec. Thing is if I just use
>>>  windows to copy the same file from one folder to the other it does
>>>  the whole file (3.7GB) in about 5.5 seconds (timed with my phone)
>>>  so that's also a pretty massive difference.
>>>
>>>  Maybe this is normal and I've just not noticed it on these other
>>>  servers since they have a much smaller amount of data to backup?
>>>  Still seems like some thing is wrong. I wouldn't expect the speed
>>>  difference to be that huge.
>>>
>>>
>>>
>>>  -- Original Message -- From: "Cary Lewis"
>>>   To: br...@sqls.net Sent: 2/10/2014 3:56:35
>>>  PM Subject: Re: Rsync performance with large exchange database
>>>  files
>>>
>>>   when you were doing rsync from /cygdrive/c to /cygdrive/d was
  the exchange file already there? Or was it clean copy?

  On Mon, Feb 10, 2014 at 4:03 PM,  wrote:

>  Okay, so I've done some testing..
>
>  I created a roughly 4gb file from one of the smaller exchange
>  database files.
>
>  If I copy that to remotely to my desktop, I get about
>  45-50MB/sec read speed off the D (exchange database) drive. If
>  I copy that back to the C drive (just the OS) for the Windows
>  server it writes to the C drive at almost 100MB/sec over the
>  network.
>
>  If I copy directly from the server D drive to it's C drive
>  using windows it's around 45MB/sec
>
>  Inside cygwin using just the copy command I get about
>  35-45MB/sec transfer speed so there's a little hit just from
>  cygwin.
>
>  Using rsync to "sync" the file from the D drive to the C drive
>  with the --progress option. I'm getting about 2-2.5MB/sec
>  transfer speed
>
>  The server is being used... So I've run the tests a few times
>  thoughout the last hour or so and these are about my average
>  numbers.
>
>  Why would rsync be so much slower? Is there something I can
>  test to help figure this out? I'm using rsync on a couple dozen
>  Windows servers and it's been working great so I'm not sure why
>  this one is acting weird.
>
>
>
>  -- Original Message -- From: "Kevin Korb"
>   To: rsync@lists.samba.org Sent: 2/10/2014
>  10:57:08 AM Subject: Re: Rsync performance with large exchange
>  database files
>
>   3.1.0 will probably help some.
>>>
>>>  What are the specs of the FreeBSD system? I have found that ZFS on
>>>  FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is
>>>  the minimum if dedup is disabled and 16BG of RAM for when dedup is
>>>
   enabled.
>>>
>>
>>>  Also, a cache disk helps a lot.
>>>
>>>  On 02/10/2014 10:22 AM, br...@sqls.net wrote:
>>>
>>>

Re[2]: Rsync performance with large exchange database files

[bruce]

Well the local test was just to a test to see if I could understand why 
the remote sync of the exchange database was so slow.  I've heard that 
rsync is less efficient for local copies but this isn't like 80% the 
performance, or half the performance.. It's a massive difference - which 
I wasn't expecting to see.


I have looked at unison but I also backup several linux and bsd systems 
here and rsync just seemed like a good choice since it's still be 
developed and it works on everything.


In the end, I'm still looking to understand why a sync of a large 
exchange database file is taking 30-40 hours to finish and if there's 
anything I can do to help reduce that window.  I can't add a cache drive 
to the FreeBSD server very easily at the moment so I was trying to 
narrow down if the issue is on the BSD side or Windows side or maybe a 
mix of both.


I wish a native windows client of rsync existed :)

-- Original Message --
From: "Kevin Korb" 
To: rsync@lists.samba.org
Sent: 2/10/2014 4:09:15 PM
Subject: Re: Rsync performance with large exchange database files


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rsync is known to be pretty inefficient on local copies (-W is forced
there btw) and cygwin doesn't really help with that either.
Essentially, when not networking rsync isn't much smarter than cp but
it has a ton of extra overhead.

Also, maybe you want unison since there is a native Windows version of 
it?


On 02/10/2014 05:05 PM, br...@sqls.net wrote:


 Clean copy. I even used the -W flag to see if it made a difference
 but, nope.

 I'm testing this same test on some of my other servers too. See if
 there's any common-ground I can find.

 On another servers (MS SQL Server) with faster disks I tried a
 similar test just now. There's only the C drive on this server but
 I used my same test file from the other server and used rsync
 (3.1.0) to copy the folder from one folder to another folder and it
 kicked off and got up to about 25MB/sec. Thing is if I just use
 windows to copy the same file from one folder to the other it does
 the whole file (3.7GB) in about 5.5 seconds (timed with my phone)
 so that's also a pretty massive difference.

 Maybe this is normal and I've just not noticed it on these other
 servers since they have a much smaller amount of data to backup?
 Still seems like some thing is wrong. I wouldn't expect the speed
 difference to be that huge.



 -- Original Message -- From: "Cary Lewis"
  To: br...@sqls.net Sent: 2/10/2014 3:56:35
 PM Subject: Re: Rsync performance with large exchange database
 files


 when you were doing rsync from /cygdrive/c to /cygdrive/d was
 the exchange file already there? Or was it clean copy?

 On Mon, Feb 10, 2014 at 4:03 PM,  wrote:

 Okay, so I've done some testing..

 I created a roughly 4gb file from one of the smaller exchange
 database files.

 If I copy that to remotely to my desktop, I get about
 45-50MB/sec read speed off the D (exchange database) drive. If
 I copy that back to the C drive (just the OS) for the Windows
 server it writes to the C drive at almost 100MB/sec over the
 network.

 If I copy directly from the server D drive to it's C drive
 using windows it's around 45MB/sec

 Inside cygwin using just the copy command I get about
 35-45MB/sec transfer speed so there's a little hit just from
 cygwin.

 Using rsync to "sync" the file from the D drive to the C drive
 with the --progress option. I'm getting about 2-2.5MB/sec
 transfer speed

 The server is being used... So I've run the tests a few times
 thoughout the last hour or so and these are about my average
 numbers.

 Why would rsync be so much slower? Is there something I can
 test to help figure this out? I'm using rsync on a couple dozen
 Windows servers and it's been working great so I'm not sure why
 this one is acting weird.



 -- Original Message -- From: "Kevin Korb"
  To: rsync@lists.samba.org Sent: 2/10/2014
 10:57:08 AM Subject: Re: Rsync performance with large exchange
 database files


 3.1.0 will probably help some.

 What are the specs of the FreeBSD system? I have found that ZFS on
 FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is
 the minimum if dedup is disabled and 16BG of RAM for when dedup is

 enabled.


 Also, a cache disk helps a lot.

 On 02/10/2014 10:22 AM, br...@sqls.net wrote:




 -- Original Message -- From: br...@sqls.net
  To: rsync@lists.samba.org
  Sent: 2/10/2014 8:38:06 AM
 Subject: Rsync performance with large exchange database
 files


 I'm using a mixture of FreeBSD w/ ZFS+snapshots and
 rsync to backup all the servers at my day job. This
 works pretty good overall but on one server it's not
 working so well :)

 We have an Exchange 2003 server with 4 separate mail
 store databases. One of them is roughly 900GB the
 others are ~200GB, ~160GB, and ~50GB. Rsync seems to
 spend a lot of time trying to find the differences in
 the files. On

Re: Rsync performance with large exchange database files

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rsync is known to be pretty inefficient on local copies (-W is forced
there btw) and cygwin doesn't really help with that either.
Essentially, when not networking rsync isn't much smarter than cp but
it has a ton of extra overhead.

Also, maybe you want unison since there is a native Windows version of it?

On 02/10/2014 05:05 PM, br...@sqls.net wrote:
> 
> Clean copy. I even used the -W flag to see if it made a difference
> but, nope.
> 
> I'm testing this same test on some of my other servers too. See if 
> there's any common-ground I can find.
> 
> On another servers (MS SQL Server) with faster disks I tried a
> similar test just now. There's only the C drive on this server but
> I used my same test file from the other server and used rsync
> (3.1.0) to copy the folder from one folder to another folder and it
> kicked off and got up to about 25MB/sec. Thing is if I just use
> windows to copy the same file from one folder to the other it does
> the whole file (3.7GB) in about 5.5 seconds (timed with my phone)
> so that's also a pretty massive difference.
> 
> Maybe this is normal and I've just not noticed it on these other
> servers since they have a much smaller amount of data to backup?
> Still seems like some thing is wrong. I wouldn't expect the speed
> difference to be that huge.
> 
> 
> 
> -- Original Message -- From: "Cary Lewis"
>  To: br...@sqls.net Sent: 2/10/2014 3:56:35
> PM Subject: Re: Rsync performance with large exchange database
> files
> 
>> when you were doing rsync from /cygdrive/c to /cygdrive/d was
>> the exchange file already there? Or was it clean copy?
>> 
>> On Mon, Feb 10, 2014 at 4:03 PM,  wrote:
>>> Okay, so I've done some testing..
>>> 
>>> I created a roughly 4gb file from one of the smaller exchange
>>> database files.
>>> 
>>> If I copy that to remotely to my desktop, I get about
>>> 45-50MB/sec read speed off the D (exchange database) drive. If
>>> I copy that back to the C drive (just the OS) for the Windows
>>> server it writes to the C drive at almost 100MB/sec over the
>>> network.
>>> 
>>> If I copy directly from the server D drive to it's C drive
>>> using windows it's around 45MB/sec
>>> 
>>> Inside cygwin using just the copy command I get about
>>> 35-45MB/sec transfer speed so there's a little hit just from
>>> cygwin.
>>> 
>>> Using rsync to "sync" the file from the D drive to the C drive
>>> with the --progress option. I'm getting about 2-2.5MB/sec
>>> transfer speed
>>> 
>>> The server is being used... So I've run the tests a few times 
>>> thoughout the last hour or so and these are about my average
>>> numbers.
>>> 
>>> Why would rsync be so much slower? Is there something I can
>>> test to help figure this out? I'm using rsync on a couple dozen
>>> Windows servers and it's been working great so I'm not sure why
>>> this one is acting weird.
>>> 
>>> 
>>> 
>>> -- Original Message -- From: "Kevin Korb"
>>>  To: rsync@lists.samba.org Sent: 2/10/2014
>>> 10:57:08 AM Subject: Re: Rsync performance with large exchange
>>> database files
>>> 
> 3.1.0 will probably help some.
> 
> What are the specs of the FreeBSD system? I have found that ZFS on 
> FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is
> the minimum if dedup is disabled and 16BG of RAM for when dedup is
> enabled.
> 
> Also, a cache disk helps a lot.
> 
> On 02/10/2014 10:22 AM, br...@sqls.net wrote:
>> 
>> 
>> 
>> -- Original Message -- From: br...@sqls.net 
>>  To: rsync@lists.samba.org 
>>  Sent: 2/10/2014 8:38:06 AM
>> Subject: Rsync performance with large exchange database
>> files
>> 
>>> I'm using a mixture of FreeBSD w/ ZFS+snapshots and
>>> rsync to backup all the servers at my day job. This
>>> works pretty good overall but on one server it's not
>>> working so well :)
>>> 
>>> We have an Exchange 2003 server with 4 separate mail
>>> store databases. One of them is roughly 900GB the
>>> others are ~200GB, ~160GB, and ~50GB. Rsync seems to
>>> spend a lot of time trying to find the differences in
>>> the files. On the Windows server where rsync is kicked
>>> off there's very little CPU or RAM usage for the rsync
>>> client. On the server rsync (rsyncd, no ssh) is using 
>>> around 70-85% of a cpu (well, half a cpu due to hyper
>>> threading). I'm using VSS on the windows server to take
>>> a snapshot and expose it then running rsync from that
>>> to avoid locking issues.
>>> 
>>> Is there anything I should check to help narrow down
>>> "problems?" or any settings I should try that could
>>> help speed things up any?
>>> 
>>> Below is the final output of the last two rsync runs to
>>> give you an idea. It's taking 30-40+ hours to finish
>>> even though it's only transferring 80-160GB of change.
>>> Right now I'm testing 

Re[2]: Rsync performance with large exchange database files

[bruce]

Clean copy. I even used the -W flag to see if it made a difference but, 
nope.


I'm testing this same test on some of my other servers too. See if 
there's any common-ground I can find.


On another servers (MS SQL Server) with faster disks I tried a similar 
test just now. There's only the C drive on this server but I used my 
same test file from the other server and used rsync (3.1.0) to copy the 
folder from one folder to another folder and it kicked off and got up to 
about 25MB/sec. Thing is if I just use windows to copy the same file 
from one folder to the other it does the whole file (3.7GB) in about 5.5 
seconds (timed with my phone) so that's also a pretty massive 
difference.


Maybe this is normal and I've just not noticed it on these other servers 
since they have a much smaller amount of data to backup? Still seems 
like some thing is wrong. I wouldn't expect the speed difference to be 
that huge.




-- Original Message --
From: "Cary Lewis" 
To: br...@sqls.net
Sent: 2/10/2014 3:56:35 PM
Subject: Re: Rsync performance with large exchange database files


when you were doing rsync from /cygdrive/c to /cygdrive/d was the
exchange file already there? Or was it clean copy?

On Mon, Feb 10, 2014 at 4:03 PM,  wrote:

  Okay, so I've done some testing..

 I created a roughly 4gb file from one of the smaller exchange 
database

 files.

 If I copy that to remotely to my desktop, I get about 45-50MB/sec 
read speed
 off the D (exchange database) drive. If I copy that back to the C 
drive
 (just the OS) for the Windows server it writes to the C drive at 
almost

 100MB/sec over the network.

 If I copy directly from the server D drive to it's C drive using 
windows

 it's around 45MB/sec

 Inside cygwin using just the copy command I get about 35-45MB/sec 
transfer

 speed so there's a little hit just from cygwin.

 Using rsync to "sync" the file from the D drive to the C drive with 
the

 --progress option. I'm getting about 2-2.5MB/sec transfer speed

 The server is being used... So I've run the tests a few times 
thoughout the

 last hour or so and these are about my average numbers.

 Why would rsync be so much slower? Is there something I can test to 
help
 figure this out? I'm using rsync on a couple dozen Windows servers 
and it's

 been working great so I'm not sure why this one is acting weird.



 -- Original Message --
 From: "Kevin Korb" 
 To: rsync@lists.samba.org
 Sent: 2/10/2014 10:57:08 AM
 Subject: Re: Rsync performance with large exchange database files


 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 3.1.0 will probably help some.

 What are the specs of the FreeBSD system? I have found that ZFS on
 FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is the
 minimum if dedup is disabled and 16BG of RAM for when dedup is 
enabled.


 Also, a cache disk helps a lot.

 On 02/10/2014 10:22 AM, br...@sqls.net wrote:




  -- Original Message -- From: br...@sqls.net
   To: rsync@lists.samba.org
   Sent: 2/10/2014 8:38:06 AM Subject:
  Rsync performance with large exchange database files


  I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to
  backup all the servers at my day job. This works pretty good
  overall but on one server it's not working so well :)

  We have an Exchange 2003 server with 4 separate mail store
  databases. One of them is roughly 900GB the others are ~200GB,
  ~160GB, and ~50GB. Rsync seems to spend a lot of time trying to
  find the differences in the files. On the Windows server where
  rsync is kicked off there's very little CPU or RAM usage for the
  rsync client. On the server rsync (rsyncd, no ssh) is using
  around 70-85% of a cpu (well, half a cpu due to hyper threading).
  I'm using VSS on the windows server to take a snapshot and expose
  it then running rsync from that to avoid locking issues.

  Is there anything I should check to help narrow down "problems?"
  or any settings I should try that could help speed things up
  any?

  Below is the final output of the last two rsync runs to give you
  an idea. It's taking 30-40+ hours to finish even though it's
  only transferring 80-160GB of change. Right now I'm testing this
  against a local rsync server so it should get pretty fast network
  performance. Eventually it will be moved to our off-site backup
  but that connection is still pretty fast (20 MBbit) and the
  backup is only hitting 800-1000 Kbytes/sec.


  Number of files: 19 Number of files transferred: 6 Total file
  size: 1265.74G bytes Total transferred file size: 1057.06G bytes
  Literal data: 160.67G bytes Matched data: 896.39G bytes File list
  size: 482 File list generation time: 0.001 seconds File list
  transfer time: 0.000 seconds Total bytes sent: 160.71G Total
  bytes received: 73.74M

  sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec total
  size is 1265.74G speedup is 7.87 [sender] _exit_cleanup(code=0,
  file=/home/lapo/

Re: Rsync performance with large exchange database files

[bruce]

 Okay, so I've done some testing..

I created a roughly 4gb file from one of the smaller exchange database 
files.


If I copy that to remotely to my desktop, I get about 45-50MB/sec read 
speed off the D (exchange database) drive.  If I copy that back to the C 
drive (just the OS) for the Windows server it writes to the C drive at 
almost 100MB/sec over the network.


If I copy directly from the server D drive to it's C drive using windows 
it's around 45MB/sec


Inside cygwin using just the copy command I get about 35-45MB/sec 
transfer speed so there's a little hit just from cygwin.


Using rsync to "sync" the file from the D drive to the C drive with the 
--progress option.  I'm getting about 2-2.5MB/sec transfer speed


The server is being used... So I've run the tests a few times thoughout 
the last hour or so and these are about my average numbers.


Why would rsync be so much slower?  Is there something I can test to 
help figure this out?  I'm using rsync on a couple dozen Windows servers 
and it's been working great so I'm not sure why this one is acting 
weird.




-- Original Message --
From: "Kevin Korb" 
To: rsync@lists.samba.org
Sent: 2/10/2014 10:57:08 AM
Subject: Re: Rsync performance with large exchange database files


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

3.1.0 will probably help some.

What are the specs of the FreeBSD system? I have found that ZFS on
FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is the
minimum if dedup is disabled and 16BG of RAM for when dedup is enabled.

Also, a cache disk helps a lot.

On 02/10/2014 10:22 AM, br...@sqls.net wrote:



 -- Original Message -- From: br...@sqls.net
  To: rsync@lists.samba.org
  Sent: 2/10/2014 8:38:06 AM Subject:
 Rsync performance with large exchange database files


 I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to
 backup all the servers at my day job. This works pretty good
 overall but on one server it's not working so well :)

 We have an Exchange 2003 server with 4 separate mail store
 databases. One of them is roughly 900GB the others are ~200GB,
 ~160GB, and ~50GB. Rsync seems to spend a lot of time trying to
 find the differences in the files. On the Windows server where
 rsync is kicked off there's very little CPU or RAM usage for the
 rsync client. On the server rsync (rsyncd, no ssh) is using
 around 70-85% of a cpu (well, half a cpu due to hyper threading).
 I'm using VSS on the windows server to take a snapshot and expose
 it then running rsync from that to avoid locking issues.

 Is there anything I should check to help narrow down "problems?"
 or any settings I should try that could help speed things up
 any?

 Below is the final output of the last two rsync runs to give you
 an idea. It's taking 30-40+ hours to finish even though it's
 only transferring 80-160GB of change. Right now I'm testing this
 against a local rsync server so it should get pretty fast network
 performance. Eventually it will be moved to our off-site backup
 but that connection is still pretty fast (20 MBbit) and the
 backup is only hitting 800-1000 Kbytes/sec.


 Number of files: 19 Number of files transferred: 6 Total file
 size: 1265.74G bytes Total transferred file size: 1057.06G bytes
 Literal data: 160.67G bytes Matched data: 896.39G bytes File list
 size: 482 File list generation time: 0.001 seconds File list
 transfer time: 0.000 seconds Total bytes sent: 160.71G Total
 bytes received: 73.74M

 sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec total
 size is 1265.74G speedup is 7.87 [sender] _exit_cleanup(code=0,
 file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c,
 line=1052): about to call exit(0)

 real 2833m1.324s user 2225m55.906s sys 45m10.015s

 Number of files: 11 Number of files transferred: 6 Total file
 size: 1268.78G bytes Total transferred file size: 1251.04G bytes
 Literal data: 83.43G bytes Matched data: 1167.61G bytes File list
 size: 216 File list generation time: 1.360 seconds File list
 transfer time: 0.000 seconds Total bytes sent: 83.48G Total bytes
 received: 87.25M

 sent 83.48G bytes received 87.25M bytes 836.85K bytes/sec total
 size is 1268.78G speedup is 15.18

 real 1745m5.647s user 1129m14.000s sys 39m58.875s

 Thanks (in advance) for the help :)


 rsync options I'm using on the client are : -rltihv --progress
 --stats --inplace --modify-window=1

 On the windows client I'm using cygwin + rsync 3.0.9 but I'm going
 to test 3.1.0 there and see if there's a difference.

 On the server it's rsync 3.1.0 running rsyncd.

 Perhaps useful bit of information :).





- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
 Kevin Korb Phone: (407) 252-6853
 Systems Administrator Internet:
 FutureQuest, Inc. ke...@futurequest.net (work)
 Orlando, Florida k...@sanitarium.net (personal)
 Web page: http://www.sanitarium.net/
 PGP public key available on web site.
~*-,._.,-*

Re: /usr/bin/ssh not found when rsync is executed within rsnapshot

[Wayne Davison]

On Mon, Feb 10, 2014 at 1:44 AM, Paul Slootman wrote:

> Besides the extraneous -e option this should work.
>

No, the later --rsh option overrides the weird "v" string, so that's not
the issue.  It appears to be that whatever compiled version of rsync he is
using doesn't allow args -- it seems to be trying to find the command using
the full string, including spaces and ssh options.  Since normal rsync
allows command args there, I don't know what is strange about his setup.

There are 2 easy solutions:

1. put what you need to run in a script and specify --rsh=/path/script.

2. put your ssh options into your ~/.ssh/config file, and stop specifying
--rsh.  If you only want that key sometimes when going to that host, you
can specify a host alias in the config.  For instance:

Host debx40-backup
Hostname debx40
User backupuser
IdentityFile /home/backupuser/.ssh/id_rsa

That even lets you omit the "backupuser@" prefix on the command, since you
told ssh to use the right user, but only if you use "debx40-backup:/" for
the destination host.  If you always want those options, just remove the
"-backup" suffix (and the Hostname line) and they will get used for every
ssh to debx40 (by name).

..wayne..
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync performance with large exchange database files

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

If possible try adding a cache disk.  It doesn't have to be anything
special.

On 02/10/2014 12:12 PM, br...@sqls.net wrote:
> 
> 
> -- Original Message -- From: "Kevin Korb"
>  To: rsync@lists.samba.org Sent: 2/10/2014
> 10:57:08 AM Subject: Re: Rsync performance with large exchange
> database files
> 
>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1
>> 
>> 3.1.0 will probably help some.
>> 
>> What are the specs of the FreeBSD system? I have found that ZFS
>> on FreeBSD is extremely RAM hungry. In my experience 8GB of RAM
>> is the minimum if dedup is disabled and 16BG of RAM for when
>> dedup is enabled.
>> 
>> Also, a cache disk helps a lot.
>> 
> 
> The FreeBSD system has 32GB of ram, dual Xeon E5620 cpus, the ZFS
> pool is a 6 disk (HGST 7.2K SATA 2TB drives) raidz2 array.  There's
> no compression or deduplication enabled on the pool.  There isn't
> any cache disks setup though.
> 
> I'm starting to do some read-performance tests on the windows
> machine to see if there's a problem there.
> 
> When watching the rsync processes I'm not always clear which end
> it's waiting on.

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlL5CPAACgkQVKC1jlbQAQer4QCeNqSxVe7b6X7xXd8B4KlMb8af
anEAn2nzM2FmiWfi21g0ixDWAoqrLmur
=2BUh
-END PGP SIGNATURE-
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re[2]: Rsync performance with large exchange database files

[bruce]

-- Original Message --
From: "Kevin Korb" 
To: rsync@lists.samba.org
Sent: 2/10/2014 10:57:08 AM
Subject: Re: Rsync performance with large exchange database files


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

3.1.0 will probably help some.

What are the specs of the FreeBSD system? I have found that ZFS on
FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is the
minimum if dedup is disabled and 16BG of RAM for when dedup is enabled.

Also, a cache disk helps a lot.



The FreeBSD system has 32GB of ram, dual Xeon E5620 cpus, the ZFS pool 
is a 6 disk (HGST 7.2K SATA 2TB drives) raidz2 array.  There's no 
compression or deduplication enabled on the pool.  There isn't any cache 
disks setup though.


I'm starting to do some read-performance tests on the windows machine to 
see if there's a problem there.


When watching the rsync processes I'm not always clear which end it's 
waiting on.


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync performance with large exchange database files

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

3.1.0 will probably help some.

What are the specs of the FreeBSD system?  I have found that ZFS on
FreeBSD is extremely RAM hungry.  In my experience 8GB of RAM is the
minimum if dedup is disabled and 16BG of RAM for when dedup is enabled.

Also, a cache disk helps a lot.

On 02/10/2014 10:22 AM, br...@sqls.net wrote:
> 
> 
> -- Original Message -- From: br...@sqls.net
>  To: rsync@lists.samba.org
>  Sent: 2/10/2014 8:38:06 AM Subject:
> Rsync performance with large exchange database files
> 
>> I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to
>> backup all the servers at my day job. This works pretty good
>> overall but on one server it's not working so well :)
>> 
>> We have an Exchange 2003 server with 4 separate mail store
>> databases. One of them is roughly 900GB the others are ~200GB,
>> ~160GB, and ~50GB. Rsync seems to spend a lot of time trying to
>> find the differences in the files. On the Windows server where
>> rsync is kicked off there's very little CPU or RAM usage for the
>> rsync client. On the server rsync (rsyncd, no ssh) is using
>> around 70-85% of a cpu (well, half a cpu due to hyper threading).
>> I'm using VSS on the windows server to take a snapshot and expose
>> it then running rsync from that to avoid locking issues.
>> 
>> Is there anything I should check to help narrow down "problems?"
>> or any settings I should try that could help speed things up
>> any?
>> 
>> Below is the final output of the last two rsync runs to give you
>> an idea. It's taking 30-40+ hours to finish even though it's
>> only transferring 80-160GB of change. Right now I'm testing this
>> against a local rsync server so it should get pretty fast network
>> performance. Eventually it will be moved to our off-site backup
>> but that connection is still pretty fast (20 MBbit) and the
>> backup is only hitting 800-1000 Kbytes/sec.
>> 
>> 
>> Number of files: 19 Number of files transferred: 6 Total file
>> size: 1265.74G bytes Total transferred file size: 1057.06G bytes 
>> Literal data: 160.67G bytes Matched data: 896.39G bytes File list
>> size: 482 File list generation time: 0.001 seconds File list
>> transfer time: 0.000 seconds Total bytes sent: 160.71G Total
>> bytes received: 73.74M
>> 
>> sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec total
>> size is 1265.74G speedup is 7.87 [sender] _exit_cleanup(code=0, 
>> file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, 
>> line=1052): about to call exit(0)
>> 
>> real 2833m1.324s user 2225m55.906s sys 45m10.015s
>> 
>> Number of files: 11 Number of files transferred: 6 Total file
>> size: 1268.78G bytes Total transferred file size: 1251.04G bytes 
>> Literal data: 83.43G bytes Matched data: 1167.61G bytes File list
>> size: 216 File list generation time: 1.360 seconds File list
>> transfer time: 0.000 seconds Total bytes sent: 83.48G Total bytes
>> received: 87.25M
>> 
>> sent 83.48G bytes received 87.25M bytes 836.85K bytes/sec total
>> size is 1268.78G speedup is 15.18
>> 
>> real 1745m5.647s user 1129m14.000s sys 39m58.875s
>> 
>> Thanks (in advance) for the help :)
> 
> rsync options I'm using on the client are : -rltihv --progress
> --stats --inplace --modify-window=1
> 
> On the windows client I'm using cygwin + rsync 3.0.9 but I'm going
> to test 3.1.0 there and see if there's a difference.
> 
> On the server it's rsync 3.1.0 running rsyncd.
> 
> Perhaps useful bit of information :).
> 
> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlL5BOQACgkQVKC1jlbQAQeEVACgl7LEtMrX52lMqNukkCn3+poR
k/oAoOdugEyjn5AeP54zvbwazhjyIKc4
=ll+E
-END PGP SIGNATURE-
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync performance with large exchange database files

[bruce]

-- Original Message --
From: br...@sqls.net
To: rsync@lists.samba.org
Sent: 2/10/2014 8:38:06 AM
Subject: Rsync performance with large exchange database files

I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to backup all 
the servers at my day job. This works pretty good overall but on one 
server it's not working so well :)


We have an Exchange 2003 server with 4 separate mail store databases. 
One of them is roughly 900GB the others are ~200GB, ~160GB, and ~50GB. 
Rsync seems to spend a lot of time trying to find the differences in 
the files. On the Windows server where rsync is kicked off there's very 
little CPU or RAM usage for the rsync client. On the server rsync 
(rsyncd, no ssh) is using around 70-85% of a cpu (well, half a cpu due 
to hyper threading). I'm using VSS on the windows server to take a 
snapshot and expose it then running rsync from that to avoid locking 
issues.


Is there anything I should check to help narrow down "problems?" or any 
settings I should try that could help speed things up any?


Below is the final output of the last two rsync runs to give you an 
idea. It's taking 30-40+ hours to finish even though it's only 
transferring 80-160GB of change. Right now I'm testing this against a 
local rsync server so it should get pretty fast network performance. 
Eventually it will be moved to our off-site backup but that connection 
is still pretty fast (20 MBbit) and the backup is only hitting 800-1000 
Kbytes/sec.



Number of files: 19
Number of files transferred: 6
Total file size: 1265.74G bytes
Total transferred file size: 1057.06G bytes
Literal data: 160.67G bytes
Matched data: 896.39G bytes
File list size: 482
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 160.71G
Total bytes received: 73.74M

sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec
total size is 1265.74G speedup is 7.87
[sender] _exit_cleanup(code=0, 
file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, 
line=1052): about to call exit(0)


real 2833m1.324s
user 2225m55.906s
sys 45m10.015s

Number of files: 11
Number of files transferred: 6
Total file size: 1268.78G bytes
Total transferred file size: 1251.04G bytes
Literal data: 83.43G bytes
Matched data: 1167.61G bytes
File list size: 216
File list generation time: 1.360 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 83.48G
Total bytes received: 87.25M

sent 83.48G bytes received 87.25M bytes 836.85K bytes/sec
total size is 1268.78G speedup is 15.18

real 1745m5.647s
user 1129m14.000s
sys 39m58.875s

Thanks (in advance) for the help :)


rsync options I'm using on the client are : -rltihv --progress --stats 
--inplace --modify-window=1


On the windows client I'm using cygwin + rsync 3.0.9 but I'm going to 
test 3.1.0 there and see if there's a difference.


On the server it's rsync 3.1.0 running rsyncd.

Perhaps useful bit of information :).
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rsync performance with large exchange database files

[bruce]

I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to backup all 
the servers at my day job. This works pretty good overall but on one 
server it's not working so well :)


We have an Exchange 2003 server with 4 separate mail store databases. 
One of them is roughly 900GB the others are ~200GB, ~160GB, and ~50GB. 
Rsync seems to spend a lot of time trying to find the differences in the 
files. On the Windows server where rsync is kicked off there's very 
little CPU or RAM usage for the rsync client. On the server rsync 
(rsyncd, no ssh) is using around 70-85% of a cpu (well, half a cpu due 
to hyper threading). I'm using VSS on the windows server to take a 
snapshot and expose it then running rsync from that to avoid locking 
issues.


Is there anything I should check to help narrow down "problems?" or any 
settings I should try that could help speed things up any?


Below is the final output of the last two rsync runs to give you an 
idea. It's taking 30-40+ hours to finish even though it's only 
transferring 80-160GB of change. Right now I'm testing this against a 
local rsync server so it should get pretty fast network performance. 
Eventually it will be moved to our off-site backup but that connection 
is still pretty fast (20 MBbit) and the backup is only hitting 800-1000 
Kbytes/sec.



Number of files: 19
Number of files transferred: 6
Total file size: 1265.74G bytes
Total transferred file size: 1057.06G bytes
Literal data: 160.67G bytes
Matched data: 896.39G bytes
File list size: 482
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 160.71G
Total bytes received: 73.74M

sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec
total size is 1265.74G speedup is 7.87
[sender] _exit_cleanup(code=0, 
file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, 
line=1052): about to call exit(0)


real 2833m1.324s
user 2225m55.906s
sys 45m10.015s

Number of files: 11
Number of files transferred: 6
Total file size: 1268.78G bytes
Total transferred file size: 1251.04G bytes
Literal data: 83.43G bytes
Matched data: 1167.61G bytes
File list size: 216
File list generation time: 1.360 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 83.48G
Total bytes received: 87.25M

sent 83.48G bytes received 87.25M bytes 836.85K bytes/sec
total size is 1268.78G speedup is 15.18

real 1745m5.647s
user 1129m14.000s
sys 39m58.875s

Thanks (in advance) for the help :)
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Bugg when using Extended Attributes flag -X

[Henri Shustak]

Hello,

One approach is to backup to a disk image on Mac OS X (.sparsebundle) and then 
to push or pull the disk image over to your remote GNU/LINUX system (possibly 
via rsync. LBackup has a scripting sub-system to handle exactly this kind of 
situation. It is not as fancy as the bug fix you proposed. However, it will 
result in your backup having various extended attributes.

This  URL provides information 
about the initialisation scripts which are designed to make configuring backup 
scenarios like the one listed above (image attaching and detaching) easier. 
Much more work is needed to extend the repository of initialisation scripts 
which are bundled with LBackup. However, the disk image pre and post hook 
initialization script which is listed on that page works a treat and saves 
time. It is just a starting point. Contributions, feedback and alliterative 
approaches to this area of LBackup are certainly welcome. 

The idea behind the initialization scripts is to simplify (via automation) a 
range of steps (often within the LBackup scripting sub-system) so that the 
setup is as painless as possible. However, ideally the initialisation scripts 
could eventually extend well beyond the scripting subsystem and into other 
areas in order to simplify the setup monitoring and verification of backups on 
multiple servers. 

Hope this helps.
Henri



This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 5/02/2014, at 12:54 AM, Sun_Blood  wrote:

> Hello,
> 
> I found that using rsync on OS X can give some problems when it comes to 
> Extended Attributes (-X flag).
> 
> The server I use has Ubuntu with the filesystem XFS and I am trying to backup 
> a OS X system to it. The problem is as far as I understand it that Linux 
> Kernel has a liming on 64k för Extended Attributes and OS X don’t have this 
> limit.
> 
> Some error output.
> rsync: rsync_xal_set: 
> lsetxattr(""/srv/danne/extern2/1000_EXT/2013/2013-03-05/IMG_6872-Edit.tif"","user.com.apple.ResourceFork")
>  failed: Argument list too long (7)
> 
> Error 2
> rsync: rsync_xal_set: 
> lsetxattr(""/srv/nas/home/apple_bak_rsync/x/Pictures/iPhoto 
> Library/Database/BigBlobs.apdb"","user.com.apple.FinderInfo") failed: 
> Operation not permitted (1)
> 
> Both this errors will go away if removing the -X flag from rsync.
> 
> What I would like to see is a feature in rsync that checks the destination 
> operation system environment and if it can’t handle the size of the EA being 
> transferred then stores the EA information in a separate file.
> 
> //Sun_Blood
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: /usr/bin/ssh not found when rsync is executed within rsnapshot

[Paul Slootman]

On Mon 10 Feb 2014, Lorenz wrote:

> grep -v "#" /etc/rsnapshot | grep [a-z]
> i.e. the /etc/rsnapshot minus the comments and the empty lines:

I'd recommend using 'grep .' to find non-empty lines... shorter and more
accurate :-)

> rsync_long_args   -ev --rsync-path=/home/backupuser/rsync-wrapper.sh

-e is the short version of --rsh so I don't know what you're trying to
do here... use the 'v' command instead of (the default) ssh? Probably not.

> /usr/bin/rsync -av -ev --rsync-path=/home/backupuser/rsync-wrapper.sh \
> --rsh="/usr/bin/ssh -i /home/backupuser/.ssh/id_rsa" backupuser@debx40:/ \
> /media/extfp/Backup/rsnapshot/test/hourly.0/debx40/
> rsync: Failed to exec /usr/bin/ssh -i /home/backupuser/.ssh/id_rsa: No such 
> file or directory (2)

Besides the extraneous -e option this should work.
The error message is a bit misleading though.

Make sure that there are no "wrong" whitespace characters in there.
I've fallen into the trap of copy&pasting commands / configs from
websites and having them fail mysteriously, until I noticed I could not
left-shift those lines in vim with << . Those leading spaces were not
spaces but "no-break spaces", hex value 0x80.

So check your config / scripts with "LANG=C cat -v /etc/rsnapshot" etc.

> What could be the reason? How could i debug this?

I often use "strace -f -e execve command ..." and / or
"strace -f -e execve command ..." in such cases to see what
it is really trying to run.

Paul
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html