Bruce there is also bacula which seems to be available for all the os's you are running.
http://www.bacula.org/en/?page=documentation On Mon, Feb 10, 2014 at 11:24 PM, <br...@sqls.net> wrote: > > Well the local test was just to a test to see if I could understand why > the remote sync of the exchange database was so slow. I've heard that > rsync is less efficient for local copies but this isn't like 80% the > performance, or half the performance.. It's a massive difference - which I > wasn't expecting to see. > > I have looked at unison but I also backup several linux and bsd systems > here and rsync just seemed like a good choice since it's still be developed > and it works on everything. > > In the end, I'm still looking to understand why a sync of a large exchange > database file is taking 30-40 hours to finish and if there's anything I can > do to help reduce that window. I can't add a cache drive to the FreeBSD > server very easily at the moment so I was trying to narrow down if the > issue is on the BSD side or Windows side or maybe a mix of both. > > I wish a native windows client of rsync existed :) > > ------ Original Message ------ > From: "Kevin Korb" <k...@sanitarium.net> > To: rsync@lists.samba.org > Sent: 2/10/2014 4:09:15 PM > Subject: Re: Rsync performance with large exchange database files > > -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Rsync is known to be pretty inefficient on local copies (-W is forced >> there btw) and cygwin doesn't really help with that either. >> Essentially, when not networking rsync isn't much smarter than cp but >> it has a ton of extra overhead. >> >> Also, maybe you want unison since there is a native Windows version of it? >> >> On 02/10/2014 05:05 PM, br...@sqls.net wrote: >> >>> >>> Clean copy. I even used the -W flag to see if it made a difference >>> but, nope. >>> >>> I'm testing this same test on some of my other servers too. See if >>> there's any common-ground I can find. >>> >>> On another servers (MS SQL Server) with faster disks I tried a >>> similar test just now. There's only the C drive on this server but >>> I used my same test file from the other server and used rsync >>> (3.1.0) to copy the folder from one folder to another folder and it >>> kicked off and got up to about 25MB/sec. Thing is if I just use >>> windows to copy the same file from one folder to the other it does >>> the whole file (3.7GB) in about 5.5 seconds (timed with my phone) >>> so that's also a pretty massive difference. >>> >>> Maybe this is normal and I've just not noticed it on these other >>> servers since they have a much smaller amount of data to backup? >>> Still seems like some thing is wrong. I wouldn't expect the speed >>> difference to be that huge. >>> >>> >>> >>> ------ Original Message ------ From: "Cary Lewis" >>> <cary.le...@gmail.com> To: br...@sqls.net Sent: 2/10/2014 3:56:35 >>> PM Subject: Re: Rsync performance with large exchange database >>> files >>> >>> when you were doing rsync from /cygdrive/c to /cygdrive/d was >>>> the exchange file already there? Or was it clean copy? >>>> >>>> On Mon, Feb 10, 2014 at 4:03 PM, <br...@sqls.net> wrote: >>>> >>>>> Okay, so I've done some testing.. >>>>> >>>>> I created a roughly 4gb file from one of the smaller exchange >>>>> database files. >>>>> >>>>> If I copy that to remotely to my desktop, I get about >>>>> 45-50MB/sec read speed off the D (exchange database) drive. If >>>>> I copy that back to the C drive (just the OS) for the Windows >>>>> server it writes to the C drive at almost 100MB/sec over the >>>>> network. >>>>> >>>>> If I copy directly from the server D drive to it's C drive >>>>> using windows it's around 45MB/sec >>>>> >>>>> Inside cygwin using just the copy command I get about >>>>> 35-45MB/sec transfer speed so there's a little hit just from >>>>> cygwin. >>>>> >>>>> Using rsync to "sync" the file from the D drive to the C drive >>>>> with the --progress option. I'm getting about 2-2.5MB/sec >>>>> transfer speed >>>>> >>>>> The server is being used... So I've run the tests a few times >>>>> thoughout the last hour or so and these are about my average >>>>> numbers. >>>>> >>>>> Why would rsync be so much slower? Is there something I can >>>>> test to help figure this out? I'm using rsync on a couple dozen >>>>> Windows servers and it's been working great so I'm not sure why >>>>> this one is acting weird. >>>>> >>>>> >>>>> >>>>> ------ Original Message ------ From: "Kevin Korb" >>>>> <k...@sanitarium.net> To: rsync@lists.samba.org Sent: 2/10/2014 >>>>> 10:57:08 AM Subject: Re: Rsync performance with large exchange >>>>> database files >>>>> >>>>> 3.1.0 will probably help some. >>> >>> What are the specs of the FreeBSD system? I have found that ZFS on >>> FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is >>> the minimum if dedup is disabled and 16BG of RAM for when dedup is >>> >>>> enabled. >>>>>>> >>>>>> >>> Also, a cache disk helps a lot. >>> >>> On 02/10/2014 10:22 AM, br...@sqls.net wrote: >>> >>>> >>>>>>>> >>>>>>>> >>>>>>>> ------ Original Message ------ From: br...@sqls.net >>>>>>>> <mailto:br...@sqls.net> To: rsync@lists.samba.org >>>>>>>> <mailto:rsync@lists.samba.org> Sent: 2/10/2014 8:38:06 AM >>>>>>>> Subject: Rsync performance with large exchange database >>>>>>>> files >>>>>>>> >>>>>>>> I'm using a mixture of FreeBSD w/ ZFS+snapshots and >>>>>>>>> rsync to backup all the servers at my day job. This >>>>>>>>> works pretty good overall but on one server it's not >>>>>>>>> working so well :) >>>>>>>>> >>>>>>>>> We have an Exchange 2003 server with 4 separate mail >>>>>>>>> store databases. One of them is roughly 900GB the >>>>>>>>> others are ~200GB, ~160GB, and ~50GB. Rsync seems to >>>>>>>>> spend a lot of time trying to find the differences in >>>>>>>>> the files. On the Windows server where rsync is kicked >>>>>>>>> off there's very little CPU or RAM usage for the rsync >>>>>>>>> client. On the server rsync (rsyncd, no ssh) is using >>>>>>>>> around 70-85% of a cpu (well, half a cpu due to hyper >>>>>>>>> threading). I'm using VSS on the windows server to take >>>>>>>>> a snapshot and expose it then running rsync from that >>>>>>>>> to avoid locking issues. >>>>>>>>> >>>>>>>>> Is there anything I should check to help narrow down >>>>>>>>> "problems?" or any settings I should try that could >>>>>>>>> help speed things up any? >>>>>>>>> >>>>>>>>> Below is the final output of the last two rsync runs to >>>>>>>>> give you an idea. It's taking 30-40+ hours to finish >>>>>>>>> even though it's only transferring 80-160GB of change. >>>>>>>>> Right now I'm testing this against a local rsync server >>>>>>>>> so it should get pretty fast network performance. >>>>>>>>> Eventually it will be moved to our off-site backup but >>>>>>>>> that connection is still pretty fast (20 MBbit) and >>>>>>>>> the backup is only hitting 800-1000 Kbytes/sec. >>>>>>>>> >>>>>>>>> >>>>>>>>> Number of files: 19 Number of files transferred: 6 >>>>>>>>> Total file size: 1265.74G bytes Total transferred file >>>>>>>>> size: 1057.06G bytes Literal data: 160.67G bytes >>>>>>>>> Matched data: 896.39G bytes File list size: 482 File >>>>>>>>> list generation time: 0.001 seconds File list transfer >>>>>>>>> time: 0.000 seconds Total bytes sent: 160.71G Total >>>>>>>>> bytes received: 73.74M >>>>>>>>> >>>>>>>>> sent 160.71G bytes received 73.74M bytes 991.84K >>>>>>>>> bytes/sec total size is 1265.74G speedup is 7.87 >>>>>>>>> [sender] _exit_cleanup(code=0, >>>>>>>>> file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, >>>>>>>>> >>>>>>>>> >>>>>>>>> line=1052): about to call exit(0) >> >>> >>>>>>>>> real 2833m1.324s user 2225m55.906s sys 45m10.015s >>>>>>>>> >>>>>>>>> Number of files: 11 Number of files transferred: 6 >>>>>>>>> Total file size: 1268.78G bytes Total transferred file >>>>>>>>> size: 1251.04G bytes Literal data: 83.43G bytes Matched >>>>>>>>> data: 1167.61G bytes File list size: 216 File list >>>>>>>>> generation time: 1.360 seconds File list transfer time: >>>>>>>>> 0.000 seconds Total bytes sent: 83.48G Total bytes >>>>>>>>> received: 87.25M >>>>>>>>> >>>>>>>>> sent 83.48G bytes received 87.25M bytes 836.85K >>>>>>>>> bytes/sec total size is 1268.78G speedup is 15.18 >>>>>>>>> >>>>>>>>> real 1745m5.647s user 1129m14.000s sys 39m58.875s >>>>>>>>> >>>>>>>>> Thanks (in advance) for the help :) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> rsync options I'm using on the client are : -rltihv >>>>>>>> --progress --stats --inplace --modify-window=1 >>>>>>>> >>>>>>>> On the windows client I'm using cygwin + rsync 3.0.9 but >>>>>>>> I'm going to test 3.1.0 there and see if there's a >>>>>>>> difference. >>>>>>>> >>>>>>>> On the server it's rsync 3.1.0 running rsyncd. >>>>>>>> >>>>>>>> Perhaps useful bit of information :). >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>> -- Please use reply-all for most replies to avoid omitting >>>>>> the mailing list. To unsubscribe or change options: >>>>>> https://lists.samba.org/mailman/listinfo/rsync Before >>>>>> posting, read: >>>>>> http://www.catb.org/~esr/faqs/smart-questions.html >>>>>> >>>>> >>>>> >>>>> -- Please use reply-all for most replies to avoid omitting the >>>>> mailing list. To unsubscribe or change options: >>>>> https://lists.samba.org/mailman/listinfo/rsync Before posting, >>>>> read: http://www.catb.org/~esr/faqs/smart-questions.html >>>>> >>>> >>> >> - -- >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~' >> `^`'~*-,._.,-*~ >> Kevin Korb Phone: (407) 252-6853 >> Systems Administrator Internet: >> FutureQuest, Inc. ke...@futurequest.net (work) >> Orlando, Florida k...@sanitarium.net (personal) >> Web page: http://www.sanitarium.net/ >> PGP public key available on web site. >> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~' >> `^`'~*-,._.,-*~ >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v2.0.22 (GNU/Linux) >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >> iEYEARECAAYFAlL5TgsACgkQVKC1jlbQAQfNTwCfUV1iYYTujpQLHQxdBD1KFLm8 >> N9kAoMlhdECrmIT/LwC0tUHgXcYMjDfI >> =42pY >> -----END PGP SIGNATURE----- >> -- >> Please use reply-all for most replies to avoid omitting the mailing list. >> To unsubscribe or change options: https://lists.samba.org/ >> mailman/listinfo/rsync >> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >> > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/ > mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -- Jonathan Aquilina
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html