On 22.01.2015 14:32, Joe wrote: > Maybe a bit off topic. (I don't deal with any data even remotely that > large.) > > How would you use these new options - just as a way to break large tasks > into smaller "batches"? > If rsync "stops in the middle", then the target would be in a sort of > limbo where it might not be fully usable.
If i would guess. The description say to me: "big fat storage system underneath". Which means: "Bandwith limited" by a singular rsync process. So: First you do a "--max-depth=2" (or so) run to set a basis. Then you can do several "-min-depth=2" runs (for different parts) in parallel, to get the blood of the storage system pumping. That also uses more than 1 CPU, which a singular rsync would be limited to. If you have a "big fat" server with a truckload of cores and an I/O-System that can do several GB/s, the about 500MB/s i get on my personal computer for a singular rsync is a limiting factor. Or just think about the upcoming PCIe NVMe SSDs that can do several GB/s (and plug several of them into a computer). You need several rsync processes to spread the I/O load over enough CPUs just for duplicating a single such SSD to another, if you want to do that in the shortest time possible. > On 01/22/2015 09:38 AM, samba-b...@samba.org wrote: > > https://bugzilla.samba.org/show_bug.cgi?id=11067 > > > > Bug ID: 11067 > > Summary: add --min-depth and --max-depth options > > Product: rsync > > Version: 3.1.1 > > Hardware: All > > OS: All > > Status: NEW > > Severity: enhancement > > Priority: P5 > > Component: core > > Assignee: way...@samba.org > > Reporter: c...@innovates.com > > QA Contact: rsync...@samba.org > > > > As file systems are getting bigger and bigger, into the multiple petabyte > > scale > > rsync is not scaling well, but is still the tool of choice when migrating > > filesystems from one storage vender to another. > > > > Adding --min-depth and --max-depth options to control what directory depth > > rsync will operate on would allow better targeted rsync processes. > > -- Matthias -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
