max file size
Hello, does anybody know what's the maximum file size (terabytes?) when using rsync with options --checksum and / or --inplace? What file sizes have been tested in reality? Are there any experiences using rsync (with --checksum and / or --inplace) for big files with several / dozens or terabytes? Thanks a lot, Heinz-Josef Claes -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
On Sun, Nov 8, 2009 at 10:21 PM, Thomas Gutzler thomas.gutz...@gmail.comwrote: I thought --include=/this_dir/ --include=/this_dir/*** would do it, but it doesn't. The exclude * seems to overwrite the include matches: [sender] hiding file this_dir/foo because of pattern * Order is important. Whatever matches first, is what takes effect. Also, *** matches both the dir and its contents, so you could use: --include='/this_dir/***' --exclude=* As long as this_dir is in the root of the transfer, that will work. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: max file size
On Mon, 2009-11-09 at 11:43 +0100, Heinz-Josef Claes wrote: does anybody know what's the maximum file size (terabytes?) when using rsync with options --checksum and / or --inplace? What file sizes have been tested in reality? Are there any experiences using rsync (with --checksum and / or --inplace) for big files with several / dozens or terabytes? I don't believe rsync has a fixed maximum size other than what can fit in 64 bits, but I can't speak to any reliability issues that might come up with extremely large files. For what purpose are you considering --checksum? In the case where the file's size hasn't changed (probably true for large image files), it will add an extra full read of the file on both sides before the transfer begins, which would be very expensive for multi-terabyte files. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: max file size
Am Montag, 9. November 2009 17:48:35 schrieb Matt McCutchen: On Mon, 2009-11-09 at 11:43 +0100, Heinz-Josef Claes wrote: does anybody know what's the maximum file size (terabytes?) when using rsync with options --checksum and / or --inplace? What file sizes have been tested in reality? Are there any experiences using rsync (with --checksum and / or --inplace) for big files with several / dozens or terabytes? I don't believe rsync has a fixed maximum size other than what can fit in 64 bits, but I can't speak to any reliability issues that might come up with extremely large files. I've read about a fix for overrun checksum buffers with more than some hundred terabytes but that was just something undefined . . . For what purpose are you considering --checksum? In the case where the file's size hasn't changed (probably true for large image files), it will add an extra full read of the file on both sides before the transfer begins, which would be very expensive for multi-terabyte files. I want to check if the following is possible: 1. transport a big block of data (several terabytes) physically from location A to location B (very long distance) via tapes (or disks). (Location A and B use different storage technologies.) When the tapes arrive in location B, the block of data has changed in location A (a program / OS is running and storing data in it). 2. shutdown application / OS in location A, rsync the delta between Location A and B online, then restart the system in location B. (Perhaps step 2 has to be done multiple times.) -- There a lots of other aspects in this scenario, but that's another story. Regards, HJC -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6881] New: --bwlimit option uses KiB/s, but is documented as (what amounts to) kB/s
https://bugzilla.samba.org/show_bug.cgi?id=6881 Summary: --bwlimit option uses KiB/s, but is documented as (what amounts to) kB/s Product: rsync Version: 3.1.0 Platform: All OS/Version: All Status: NEW Severity: trivial Priority: P3 Component: core AssignedTo: way...@samba.org ReportedBy: rlaa...@wiktel.com QAContact: rsync...@samba.org The --bwlimit option seems to use KiB/s, as io.c's sleep_for_bwlimit() function divides by 1024. It's documented as KBPS, KBytes per second, and kilobytes per second. I'm going to attach a patch which standardizes all of this as KiB/s and kibibytes per second, to match the actual usage. Given that this is a network transfer rate, it'd be more proper (and consistent with other applications) to change the function to work in SI kilobytes per second (i.e. use 1000 instead of 1024), but that's backwards-incompatible. If you'd like to go this route, I can prepare a patch to that effect. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6881] --bwlimit option uses KiB/s, but is documented as (what amounts to) kB/s
https://bugzilla.samba.org/show_bug.cgi?id=6881 --- Comment #1 from rlaa...@wiktel.com 2009-11-09 12:54 CST --- Created an attachment (id=4934) -- (https://bugzilla.samba.org/attachment.cgi?id=4934action=view) A patch to change the documentation to use KiB/s and kibibytes per second. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: DO NOT REPLY [Bug 6881] New: --bwlimit option uses KiB/s, but is documented as (what amounts to) kB/s
samba-b...@samba.org wrote: Given that this is a network transfer rate, it'd be more proper (and consistent with other applications) to change the function to work in SI kilobytes per second (i.e. use 1000 instead of 1024), but that's backwards-incompatible. If you'd like to go this route, I can prepare a patch to that effect. I agree, SI kilobytes are more appropriate for network transfer rates. I doubt if anyone seriously depends on the 2.4% difference or even noticed it, and would encourage changing the code to SI kilos rather then the documentation to kibis. -- Jamie -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
Thanks everyone for your help, I've got what I want. Wayne Davison wrote: On Sun, Nov 8, 2009 at 10:21 PM, Thomas Gutzler thomas.gutz...@gmail.com mailto:thomas.gutz...@gmail.com wrote: I thought --include=/this_dir/ --include=/this_dir/*** would do it, but it doesn't. The exclude * seems to overwrite the include matches: [sender] hiding file this_dir/foo because of pattern * Order is important. Whatever matches first, is what takes effect. Also, *** matches both the dir and its contents, so you could use: --include='/this_dir/***' --exclude=* It seems to be very picky about the order. Thanks for pointing that out. My first attempt has been --include=*/ --include=*.foo --include=*.bar --include=/this_dir/*** --exclude=* which did nothing than *.foo and *.bar. Shuffling it around, I found that --include=*/ --include=/this_dir/*** --include=*.foo --include=*.bar --exclude=* does what I want and it even makes sense. Cheers, Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
On Tue, 2009-11-10 at 09:45 +0800, Thomas Gutzler wrote: Thanks everyone for your help, I've got what I want. Wayne Davison wrote: On Sun, Nov 8, 2009 at 10:21 PM, Thomas Gutzler thomas.gutz...@gmail.com mailto:thomas.gutz...@gmail.com wrote: I thought --include=/this_dir/ --include=/this_dir/*** would do it, but it doesn't. The exclude * seems to overwrite the include matches: [sender] hiding file this_dir/foo because of pattern * Order is important. Whatever matches first, is what takes effect. Also, *** matches both the dir and its contents, so you could use: --include='/this_dir/***' --exclude=* It seems to be very picky about the order. Thanks for pointing that out. My first attempt has been --include=*/ --include=*.foo --include=*.bar --include=/this_dir/*** --exclude=* which did nothing than *.foo and *.bar. Shuffling it around, I found that --include=*/ --include=/this_dir/*** --include=*.foo --include=*.bar --exclude=* does what I want and it even makes sense. Those two commands should be equivalent. If you have a reproducible case in which they aren't, please share it and we can see if there's a bug. I tried the first command and it worked fine for me. That is, after I fixed a typo I made in the name of this_dir, which left the --include=/this_dir/*** nonfunctional and gave a result like the one you cited above: [sender] hiding file this-dir/one because of pattern * -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
Matt McCutchen wrote: On Tue, 2009-11-10 at 09:45 +0800, Thomas Gutzler wrote: Thanks everyone for your help, I've got what I want. My first attempt has been --include=*/ --include=*.foo --include=*.bar --include=/this_dir/*** --exclude=* which did nothing than *.foo and *.bar. Shuffling it around, I found that --include=*/ --include=/this_dir/*** --include=*.foo --include=*.bar --exclude=* does what I want and it even makes sense. Those two commands should be equivalent. If you have a reproducible case in which they aren't, please share it and we can see if there's a bug. Going through my history, I found that I must have accidentally put a whitespace between '--include' and '=' which wasn't very obvious at the time thanks to the line wrapping of my terminal. I should have used an include file instead :) So no, no bugs. Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
does it make sense to run rsync over ftp (curlftpfs)?
Hi, It seems to work but very slowly. I guess it's because rsync has to read the complete file content on the remote host, so does it make any sense at all to do it over FTP? - -- Kent Tong Wicket tutorials freely available at http://www.agileskills2.org/EWDW Axis2 tutorials freely available at http://www.agileskills2.org/DWSAA -- View this message in context: http://old.nabble.com/does-it-make-sense-to-run-rsync-over-ftp-%28curlftpfs%29--tp26278059p26278059.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: does it make sense to run rsync over ftp (curlftpfs)?
Hello, Like on any other mounting system, if you run rsync over a mounted ftp volume, it won't be able to save you a lot of bandwidth with the delta algorithm as it's not running on both sides of the link. In another hand, if bandwidth is not your problem, with this method rsync should (to be confirmed by an expert, what I am not) still use incremental updates on your files if your ftp volume support append or other file modification function that rsync needs. Running rsync on both ends is the most optimized way to do it. But I feel like you don't have the choice of the other side setup. Cheers, Vitorio Le 10 nov. 09 à 05:14, Kent Tong a écrit : Hi, It seems to work but very slowly. I guess it's because rsync has to read the complete file content on the remote host, so does it make any sense at all to do it over FTP? - -- Kent Tong Wicket tutorials freely available at http://www.agileskills2.org/EWDW Axis2 tutorials freely available at http://www.agileskills2.org/DWSAA -- View this message in context: http://old.nabble.com/does-it-make-sense-to-run-rsync-over-ftp-%28curlftpfs%29--tp26278059p26278059.html Sent from the Samba - rsync mailing list archive at Nabble.com. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
