-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Rsync is known to be pretty inefficient on local copies (-W is forced there btw) and cygwin doesn't really help with that either. Essentially, when not networking rsync isn't much smarter than cp but it has a ton of extra overhead.
Also, maybe you want unison since there is a native Windows version of it? On 02/10/2014 05:05 PM, br...@sqls.net wrote: > > Clean copy. I even used the -W flag to see if it made a difference > but, nope. > > I'm testing this same test on some of my other servers too. See if > there's any common-ground I can find. > > On another servers (MS SQL Server) with faster disks I tried a > similar test just now. There's only the C drive on this server but > I used my same test file from the other server and used rsync > (3.1.0) to copy the folder from one folder to another folder and it > kicked off and got up to about 25MB/sec. Thing is if I just use > windows to copy the same file from one folder to the other it does > the whole file (3.7GB) in about 5.5 seconds (timed with my phone) > so that's also a pretty massive difference. > > Maybe this is normal and I've just not noticed it on these other > servers since they have a much smaller amount of data to backup? > Still seems like some thing is wrong. I wouldn't expect the speed > difference to be that huge. > > > > ------ Original Message ------ From: "Cary Lewis" > <cary.le...@gmail.com> To: br...@sqls.net Sent: 2/10/2014 3:56:35 > PM Subject: Re: Rsync performance with large exchange database > files > >> when you were doing rsync from /cygdrive/c to /cygdrive/d was >> the exchange file already there? Or was it clean copy? >> >> On Mon, Feb 10, 2014 at 4:03 PM, <br...@sqls.net> wrote: >>> Okay, so I've done some testing.. >>> >>> I created a roughly 4gb file from one of the smaller exchange >>> database files. >>> >>> If I copy that to remotely to my desktop, I get about >>> 45-50MB/sec read speed off the D (exchange database) drive. If >>> I copy that back to the C drive (just the OS) for the Windows >>> server it writes to the C drive at almost 100MB/sec over the >>> network. >>> >>> If I copy directly from the server D drive to it's C drive >>> using windows it's around 45MB/sec >>> >>> Inside cygwin using just the copy command I get about >>> 35-45MB/sec transfer speed so there's a little hit just from >>> cygwin. >>> >>> Using rsync to "sync" the file from the D drive to the C drive >>> with the --progress option. I'm getting about 2-2.5MB/sec >>> transfer speed >>> >>> The server is being used... So I've run the tests a few times >>> thoughout the last hour or so and these are about my average >>> numbers. >>> >>> Why would rsync be so much slower? Is there something I can >>> test to help figure this out? I'm using rsync on a couple dozen >>> Windows servers and it's been working great so I'm not sure why >>> this one is acting weird. >>> >>> >>> >>> ------ Original Message ------ From: "Kevin Korb" >>> <k...@sanitarium.net> To: rsync@lists.samba.org Sent: 2/10/2014 >>> 10:57:08 AM Subject: Re: Rsync performance with large exchange >>> database files >>> > 3.1.0 will probably help some. > > What are the specs of the FreeBSD system? I have found that ZFS on > FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is > the minimum if dedup is disabled and 16BG of RAM for when dedup is >>>>> enabled. > > Also, a cache disk helps a lot. > > On 02/10/2014 10:22 AM, br...@sqls.net wrote: >>>>>> >>>>>> >>>>>> >>>>>> ------ Original Message ------ From: br...@sqls.net >>>>>> <mailto:br...@sqls.net> To: rsync@lists.samba.org >>>>>> <mailto:rsync@lists.samba.org> Sent: 2/10/2014 8:38:06 AM >>>>>> Subject: Rsync performance with large exchange database >>>>>> files >>>>>> >>>>>>> I'm using a mixture of FreeBSD w/ ZFS+snapshots and >>>>>>> rsync to backup all the servers at my day job. This >>>>>>> works pretty good overall but on one server it's not >>>>>>> working so well :) >>>>>>> >>>>>>> We have an Exchange 2003 server with 4 separate mail >>>>>>> store databases. One of them is roughly 900GB the >>>>>>> others are ~200GB, ~160GB, and ~50GB. Rsync seems to >>>>>>> spend a lot of time trying to find the differences in >>>>>>> the files. On the Windows server where rsync is kicked >>>>>>> off there's very little CPU or RAM usage for the rsync >>>>>>> client. On the server rsync (rsyncd, no ssh) is using >>>>>>> around 70-85% of a cpu (well, half a cpu due to hyper >>>>>>> threading). I'm using VSS on the windows server to take >>>>>>> a snapshot and expose it then running rsync from that >>>>>>> to avoid locking issues. >>>>>>> >>>>>>> Is there anything I should check to help narrow down >>>>>>> "problems?" or any settings I should try that could >>>>>>> help speed things up any? >>>>>>> >>>>>>> Below is the final output of the last two rsync runs to >>>>>>> give you an idea. It's taking 30-40+ hours to finish >>>>>>> even though it's only transferring 80-160GB of change. >>>>>>> Right now I'm testing this against a local rsync server >>>>>>> so it should get pretty fast network performance. >>>>>>> Eventually it will be moved to our off-site backup but >>>>>>> that connection is still pretty fast (20 MBbit) and >>>>>>> the backup is only hitting 800-1000 Kbytes/sec. >>>>>>> >>>>>>> >>>>>>> Number of files: 19 Number of files transferred: 6 >>>>>>> Total file size: 1265.74G bytes Total transferred file >>>>>>> size: 1057.06G bytes Literal data: 160.67G bytes >>>>>>> Matched data: 896.39G bytes File list size: 482 File >>>>>>> list generation time: 0.001 seconds File list transfer >>>>>>> time: 0.000 seconds Total bytes sent: 160.71G Total >>>>>>> bytes received: 73.74M >>>>>>> >>>>>>> sent 160.71G bytes received 73.74M bytes 991.84K >>>>>>> bytes/sec total size is 1265.74G speedup is 7.87 >>>>>>> [sender] _exit_cleanup(code=0, >>>>>>> file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, >>>>>>> >>>>>>> line=1052): about to call exit(0) >>>>>>> >>>>>>> real 2833m1.324s user 2225m55.906s sys 45m10.015s >>>>>>> >>>>>>> Number of files: 11 Number of files transferred: 6 >>>>>>> Total file size: 1268.78G bytes Total transferred file >>>>>>> size: 1251.04G bytes Literal data: 83.43G bytes Matched >>>>>>> data: 1167.61G bytes File list size: 216 File list >>>>>>> generation time: 1.360 seconds File list transfer time: >>>>>>> 0.000 seconds Total bytes sent: 83.48G Total bytes >>>>>>> received: 87.25M >>>>>>> >>>>>>> sent 83.48G bytes received 87.25M bytes 836.85K >>>>>>> bytes/sec total size is 1268.78G speedup is 15.18 >>>>>>> >>>>>>> real 1745m5.647s user 1129m14.000s sys 39m58.875s >>>>>>> >>>>>>> Thanks (in advance) for the help :) >>>>>> >>>>>> >>>>>> rsync options I'm using on the client are : -rltihv >>>>>> --progress --stats --inplace --modify-window=1 >>>>>> >>>>>> On the windows client I'm using cygwin + rsync 3.0.9 but >>>>>> I'm going to test 3.1.0 there and see if there's a >>>>>> difference. >>>>>> >>>>>> On the server it's rsync 3.1.0 running rsyncd. >>>>>> >>>>>> Perhaps useful bit of information :). >>>>>> >>>>>> >>>>>> > >>>> -- Please use reply-all for most replies to avoid omitting >>>> the mailing list. To unsubscribe or change options: >>>> https://lists.samba.org/mailman/listinfo/rsync Before >>>> posting, read: >>>> http://www.catb.org/~esr/faqs/smart-questions.html >>> >>> >>> -- Please use reply-all for most replies to avoid omitting the >>> mailing list. To unsubscribe or change options: >>> https://lists.samba.org/mailman/listinfo/rsync Before posting, >>> read: http://www.catb.org/~esr/faqs/smart-questions.html > - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Florida k...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlL5TgsACgkQVKC1jlbQAQfNTwCfUV1iYYTujpQLHQxdBD1KFLm8 N9kAoMlhdECrmIT/LwC0tUHgXcYMjDfI =42pY -----END PGP SIGNATURE----- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html