Bruce there is also bacula which seems to be available for all the os's you
are running.

http://www.bacula.org/en/?page=documentation


On Mon, Feb 10, 2014 at 11:24 PM, <br...@sqls.net> wrote:

>
> Well the local test was just to a test to see if I could understand why
> the remote sync of the exchange database was so slow.  I've heard that
> rsync is less efficient for local copies but this isn't like 80% the
> performance, or half the performance.. It's a massive difference - which I
> wasn't expecting to see.
>
> I have looked at unison but I also backup several linux and bsd systems
> here and rsync just seemed like a good choice since it's still be developed
> and it works on everything.
>
> In the end, I'm still looking to understand why a sync of a large exchange
> database file is taking 30-40 hours to finish and if there's anything I can
> do to help reduce that window.  I can't add a cache drive to the FreeBSD
> server very easily at the moment so I was trying to narrow down if the
> issue is on the BSD side or Windows side or maybe a mix of both.
>
> I wish a native windows client of rsync existed :)
>
> ------ Original Message ------
> From: "Kevin Korb" <k...@sanitarium.net>
> To: rsync@lists.samba.org
> Sent: 2/10/2014 4:09:15 PM
> Subject: Re: Rsync performance with large exchange database files
>
>  -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Rsync is known to be pretty inefficient on local copies (-W is forced
>> there btw) and cygwin doesn't really help with that either.
>> Essentially, when not networking rsync isn't much smarter than cp but
>> it has a ton of extra overhead.
>>
>> Also, maybe you want unison since there is a native Windows version of it?
>>
>> On 02/10/2014 05:05 PM, br...@sqls.net wrote:
>>
>>>
>>>  Clean copy. I even used the -W flag to see if it made a difference
>>>  but, nope.
>>>
>>>  I'm testing this same test on some of my other servers too. See if
>>>  there's any common-ground I can find.
>>>
>>>  On another servers (MS SQL Server) with faster disks I tried a
>>>  similar test just now. There's only the C drive on this server but
>>>  I used my same test file from the other server and used rsync
>>>  (3.1.0) to copy the folder from one folder to another folder and it
>>>  kicked off and got up to about 25MB/sec. Thing is if I just use
>>>  windows to copy the same file from one folder to the other it does
>>>  the whole file (3.7GB) in about 5.5 seconds (timed with my phone)
>>>  so that's also a pretty massive difference.
>>>
>>>  Maybe this is normal and I've just not noticed it on these other
>>>  servers since they have a much smaller amount of data to backup?
>>>  Still seems like some thing is wrong. I wouldn't expect the speed
>>>  difference to be that huge.
>>>
>>>
>>>
>>>  ------ Original Message ------ From: "Cary Lewis"
>>>  <cary.le...@gmail.com> To: br...@sqls.net Sent: 2/10/2014 3:56:35
>>>  PM Subject: Re: Rsync performance with large exchange database
>>>  files
>>>
>>>   when you were doing rsync from /cygdrive/c to /cygdrive/d was
>>>>  the exchange file already there? Or was it clean copy?
>>>>
>>>>  On Mon, Feb 10, 2014 at 4:03 PM, <br...@sqls.net> wrote:
>>>>
>>>>>  Okay, so I've done some testing..
>>>>>
>>>>>  I created a roughly 4gb file from one of the smaller exchange
>>>>>  database files.
>>>>>
>>>>>  If I copy that to remotely to my desktop, I get about
>>>>>  45-50MB/sec read speed off the D (exchange database) drive. If
>>>>>  I copy that back to the C drive (just the OS) for the Windows
>>>>>  server it writes to the C drive at almost 100MB/sec over the
>>>>>  network.
>>>>>
>>>>>  If I copy directly from the server D drive to it's C drive
>>>>>  using windows it's around 45MB/sec
>>>>>
>>>>>  Inside cygwin using just the copy command I get about
>>>>>  35-45MB/sec transfer speed so there's a little hit just from
>>>>>  cygwin.
>>>>>
>>>>>  Using rsync to "sync" the file from the D drive to the C drive
>>>>>  with the --progress option. I'm getting about 2-2.5MB/sec
>>>>>  transfer speed
>>>>>
>>>>>  The server is being used... So I've run the tests a few times
>>>>>  thoughout the last hour or so and these are about my average
>>>>>  numbers.
>>>>>
>>>>>  Why would rsync be so much slower? Is there something I can
>>>>>  test to help figure this out? I'm using rsync on a couple dozen
>>>>>  Windows servers and it's been working great so I'm not sure why
>>>>>  this one is acting weird.
>>>>>
>>>>>
>>>>>
>>>>>  ------ Original Message ------ From: "Kevin Korb"
>>>>>  <k...@sanitarium.net> To: rsync@lists.samba.org Sent: 2/10/2014
>>>>>  10:57:08 AM Subject: Re: Rsync performance with large exchange
>>>>>  database files
>>>>>
>>>>>   3.1.0 will probably help some.
>>>
>>>  What are the specs of the FreeBSD system? I have found that ZFS on
>>>  FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is
>>>  the minimum if dedup is disabled and 16BG of RAM for when dedup is
>>>
>>>>   enabled.
>>>>>>>
>>>>>>
>>>  Also, a cache disk helps a lot.
>>>
>>>  On 02/10/2014 10:22 AM, br...@sqls.net wrote:
>>>
>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  ------ Original Message ------ From: br...@sqls.net
>>>>>>>>  <mailto:br...@sqls.net> To: rsync@lists.samba.org
>>>>>>>>  <mailto:rsync@lists.samba.org> Sent: 2/10/2014 8:38:06 AM
>>>>>>>>  Subject: Rsync performance with large exchange database
>>>>>>>>  files
>>>>>>>>
>>>>>>>>   I'm using a mixture of FreeBSD w/ ZFS+snapshots and
>>>>>>>>>  rsync to backup all the servers at my day job. This
>>>>>>>>>  works pretty good overall but on one server it's not
>>>>>>>>>  working so well :)
>>>>>>>>>
>>>>>>>>>  We have an Exchange 2003 server with 4 separate mail
>>>>>>>>>  store databases. One of them is roughly 900GB the
>>>>>>>>>  others are ~200GB, ~160GB, and ~50GB. Rsync seems to
>>>>>>>>>  spend a lot of time trying to find the differences in
>>>>>>>>>  the files. On the Windows server where rsync is kicked
>>>>>>>>>  off there's very little CPU or RAM usage for the rsync
>>>>>>>>>  client. On the server rsync (rsyncd, no ssh) is using
>>>>>>>>>  around 70-85% of a cpu (well, half a cpu due to hyper
>>>>>>>>>  threading). I'm using VSS on the windows server to take
>>>>>>>>>  a snapshot and expose it then running rsync from that
>>>>>>>>>  to avoid locking issues.
>>>>>>>>>
>>>>>>>>>  Is there anything I should check to help narrow down
>>>>>>>>>  "problems?" or any settings I should try that could
>>>>>>>>>  help speed things up any?
>>>>>>>>>
>>>>>>>>>  Below is the final output of the last two rsync runs to
>>>>>>>>>  give you an idea. It's taking 30-40+ hours to finish
>>>>>>>>>  even though it's only transferring 80-160GB of change.
>>>>>>>>>  Right now I'm testing this against a local rsync server
>>>>>>>>>  so it should get pretty fast network performance.
>>>>>>>>>  Eventually it will be moved to our off-site backup but
>>>>>>>>>  that connection is still pretty fast (20 MBbit) and
>>>>>>>>>  the backup is only hitting 800-1000 Kbytes/sec.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Number of files: 19 Number of files transferred: 6
>>>>>>>>>  Total file size: 1265.74G bytes Total transferred file
>>>>>>>>>  size: 1057.06G bytes Literal data: 160.67G bytes
>>>>>>>>>  Matched data: 896.39G bytes File list size: 482 File
>>>>>>>>>  list generation time: 0.001 seconds File list transfer
>>>>>>>>>  time: 0.000 seconds Total bytes sent: 160.71G Total
>>>>>>>>>  bytes received: 73.74M
>>>>>>>>>
>>>>>>>>>  sent 160.71G bytes received 73.74M bytes 991.84K
>>>>>>>>>  bytes/sec total size is 1265.74G speedup is 7.87
>>>>>>>>>  [sender] _exit_cleanup(code=0,
>>>>>>>>>  file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  line=1052): about to call exit(0)
>>
>>>
>>>>>>>>>  real 2833m1.324s user 2225m55.906s sys 45m10.015s
>>>>>>>>>
>>>>>>>>>  Number of files: 11 Number of files transferred: 6
>>>>>>>>>  Total file size: 1268.78G bytes Total transferred file
>>>>>>>>>  size: 1251.04G bytes Literal data: 83.43G bytes Matched
>>>>>>>>>  data: 1167.61G bytes File list size: 216 File list
>>>>>>>>>  generation time: 1.360 seconds File list transfer time:
>>>>>>>>>  0.000 seconds Total bytes sent: 83.48G Total bytes
>>>>>>>>>  received: 87.25M
>>>>>>>>>
>>>>>>>>>  sent 83.48G bytes received 87.25M bytes 836.85K
>>>>>>>>>  bytes/sec total size is 1268.78G speedup is 15.18
>>>>>>>>>
>>>>>>>>>  real 1745m5.647s user 1129m14.000s sys 39m58.875s
>>>>>>>>>
>>>>>>>>>  Thanks (in advance) for the help :)
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  rsync options I'm using on the client are : -rltihv
>>>>>>>>  --progress --stats --inplace --modify-window=1
>>>>>>>>
>>>>>>>>  On the windows client I'm using cygwin + rsync 3.0.9 but
>>>>>>>>  I'm going to test 3.1.0 there and see if there's a
>>>>>>>>  difference.
>>>>>>>>
>>>>>>>>  On the server it's rsync 3.1.0 running rsyncd.
>>>>>>>>
>>>>>>>>  Perhaps useful bit of information :).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>   -- Please use reply-all for most replies to avoid omitting
>>>>>>  the mailing list. To unsubscribe or change options:
>>>>>>  https://lists.samba.org/mailman/listinfo/rsync Before
>>>>>>  posting, read:
>>>>>>  http://www.catb.org/~esr/faqs/smart-questions.html
>>>>>>
>>>>>
>>>>>
>>>>>  -- Please use reply-all for most replies to avoid omitting the
>>>>>  mailing list. To unsubscribe or change options:
>>>>>  https://lists.samba.org/mailman/listinfo/rsync Before posting,
>>>>>  read: http://www.catb.org/~esr/faqs/smart-questions.html
>>>>>
>>>>
>>>
>> - --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'
>> `^`'~*-,._.,-*~
>>  Kevin Korb Phone: (407) 252-6853
>>  Systems Administrator Internet:
>>  FutureQuest, Inc. ke...@futurequest.net (work)
>>  Orlando, Florida k...@sanitarium.net (personal)
>>  Web page: http://www.sanitarium.net/
>>  PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'
>> `^`'~*-,._.,-*~
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.22 (GNU/Linux)
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iEYEARECAAYFAlL5TgsACgkQVKC1jlbQAQfNTwCfUV1iYYTujpQLHQxdBD1KFLm8
>> N9kAoMlhdECrmIT/LwC0tUHgXcYMjDfI
>> =42pY
>> -----END PGP SIGNATURE-----
>> --
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options: https://lists.samba.org/
>> mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/
> mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>



-- 
Jonathan Aquilina
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Reply via email to