Re: rsync taking a while on really large files
On Fri 15 Jan 2010, David Trammell wrote: > > I saw the -W option, but I wasn't sure about how it behaves as the > man pages don't have many details, and I thought there might be > other options I missed. For -W the man page just says "copy files > whole (w/o delta-xfer algorithm)" Take a moment to properly read the manpage, the above indicates you've never gone beyond the option summary. Further on, the options are discussed in detail: -W, --whole-file With this option rsync’s delta-transfer algorithm is not used and the whole file is sent as-is instead. The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the "disk" is actually a networked filesystem). This is the default when both the source and des‐ tination are specified as local paths, but only if no batch- writing option is in effect. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync taking a while on really large files
On 01/15/2010 07:46 PM, David Trammell wrote: I could use an ordinary copying script for the mail files, but I figured if rsync can do it in some more optimal way, I'll stick with it for simplicity (since it's working great for the several hundred gigs of user files). I saw the -W option, but I wasn't sure about how it behaves as the man pages don't have many details, and I thought there might be other options I missed. For -W the man page just says "copy files whole (w/o delta-xfer algorithm)" Does that mean it will copy all files with no comparison, or does it at least verify that there is some change to the file first? I suppose either way I can test it to see, which is faster, but if someone can clarify the behavior I'd appreciate it. Thanks, David Without -W set it will check for file date/time and I think size. - Original Message - From: To: Sent: Friday, January 15, 2010 12:40 PM Subject: Re: rsync taking a while on really large files On 01/15/2010 07:22 PM, David Trammell wrote: Can anyone suggest a good way to speed up rsync on really large files? In particular, when I rsync the mail spool directory, I have a few users with inboxes over 1GB and up and it seems to take a very long time to just compare the files. Maybe it would be faster to copy from scratch for files over a certain size or something if the time stamps don't match. David rsync is meant to safe bandwidth, that's the main use of the tool. If you have enough bandwidth, rsync without options might not be how you want to use it. Their is -W for whole files. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync taking a while on really large files
I could use an ordinary copying script for the mail files, but I figured if rsync can do it in some more optimal way, I'll stick with it for simplicity (since it's working great for the several hundred gigs of user files). I saw the -W option, but I wasn't sure about how it behaves as the man pages don't have many details, and I thought there might be other options I missed. For -W the man page just says "copy files whole (w/o delta-xfer algorithm)" Does that mean it will copy all files with no comparison, or does it at least verify that there is some change to the file first? I suppose either way I can test it to see, which is faster, but if someone can clarify the behavior I'd appreciate it. Thanks, David - Original Message - From: To: Sent: Friday, January 15, 2010 12:40 PM Subject: Re: rsync taking a while on really large files On 01/15/2010 07:22 PM, David Trammell wrote: Can anyone suggest a good way to speed up rsync on really large files? In particular, when I rsync the mail spool directory, I have a few users with inboxes over 1GB and up and it seems to take a very long time to just compare the files. Maybe it would be faster to copy from scratch for files over a certain size or something if the time stamps don't match. David rsync is meant to safe bandwidth, that's the main use of the tool. If you have enough bandwidth, rsync without options might not be how you want to use it. Their is -W for whole files. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync taking a while on really large files
On 01/15/2010 07:22 PM, David Trammell wrote: Can anyone suggest a good way to speed up rsync on really large files? In particular, when I rsync the mail spool directory, I have a few users with inboxes over 1GB and up and it seems to take a very long time to just compare the files. Maybe it would be faster to copy from scratch for files over a certain size or something if the time stamps don't match. David rsync is meant to safe bandwidth, that's the main use of the tool. If you have enough bandwidth, rsync without options might not be how you want to use it. Their is -W for whole files. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
