Speeding up rsync
Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Tuesday, August 24, 2021 9:40 PM, Kevin Korb wrote: > Most of the speedup is due to ssh being bound to a single CPU. You will > get a little more speed out of a single CPU system but the speedup is > unlikely to go past 2 in parallel. Even with that you may end up adding > more latency via disk seeks to accomplish much. I could run multiple instances of rsync for different parts of the transfer, and throw them to multiple backup hosts. > On 8/24/21 5:12 PM, hancooper wrote: > > > ‐‐‐ Original Message ‐‐‐ > > On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync > > rsync@lists.samba.org wrote: > > > > > In my experience backing up multiple hosts to a single host can speed > > > things up. Especially if using rsync over ssh with multiple CPUs on the > > > single host. You would need to do some experimentation to determine the > > > best number for your hardware and network. Also, if you exceed that > > > number you can put more stress on the backup machine lessening the > > > impact to the hosts being backed up. > > > On 8/24/21 3:58 PM, hancooper via rsync wrote: > > > > > > > It makes sense to we to run multiple rsync commands to speed up a > > > > backup. > > > > At work, some have argued that if I sync the data all to the same host, > > > > there is no advantage > > > > in parallelization. Whether you sync 3x 1G in parallel or 1x 3G makes > > > > no noticeable difference. > > > > Speed in general does not increase by increasing the number of sync > > > > processes. > > > > However, if one want to sync to multiple hosts, the effort may be > > > > worthwhile. > > > > What is the current advice about speeding up data transfer ? > > > > > > -- > > > Kevin Korb Phone: (407) 252-6853 > > > > Hi Kevin. What about a single host to another with multiple CPU's ? > > -- > > ~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._., > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. ke...@futurequest.net (work) > Orlando, Florida k...@sanitarium.net (personal) > Web page: https://sanitarium.net/ > PGP public key available on web site. > ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Speeding up rsync
Most of the speedup is due to ssh being bound to a single CPU. You will get a little more speed out of a single CPU system but the speedup is unlikely to go past 2 in parallel. Even with that you may end up adding more latency via disk seeks to accomplish much. On 8/24/21 5:12 PM, hancooper wrote: > ‐‐‐ Original Message ‐‐‐ > On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync > wrote: > >> In my experience backing up multiple hosts to a single host can speed >> things up. Especially if using rsync over ssh with multiple CPUs on the >> single host. You would need to do some experimentation to determine the >> best number for your hardware and network. Also, if you exceed that >> number you can put more stress on the backup machine lessening the >> impact to the hosts being backed up. >> >> On 8/24/21 3:58 PM, hancooper via rsync wrote: >> >>> It makes sense to we to run multiple rsync commands to speed up a backup. >>> At work, some have argued that if I sync the data all to the same host, >>> there is no advantage >>> in parallelization. Whether you sync 3x 1G in parallel or 1x 3G makes >>> no noticeable difference. >>> Speed in general does not increase by increasing the number of sync >>> processes. >>> However, if one want to sync to multiple hosts, the effort may be >>> worthwhile. >>> What is the current advice about speeding up data transfer ? >> >> -- >> Kevin Korb Phone: (407) 252-6853 > > > Hi Kevin. What about a single host to another with multiple CPU's ? > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Speeding up rsync
‐‐‐ Original Message ‐‐‐ On Tuesday, August 24, 2021 9:12 PM, hancooper via rsync wrote: > ‐‐‐ Original Message ‐‐‐ > On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync > rsync@lists.samba.org wrote: > > > In my experience backing up multiple hosts to a single host can speed > > things up. Especially if using rsync over ssh with multiple CPUs on the > > single host. You would need to do some experimentation to determine the > > best number for your hardware and network. Also, if you exceed that > > number you can put more stress on the backup machine lessening the > > impact to the hosts being backed up. > > On 8/24/21 3:58 PM, hancooper via rsync wrote: > > > > > It makes sense to we to run multiple rsync commands to speed up a backup. > > > At work, some have argued that if I sync the data all to the same host, > > > there is no advantage > > > in parallelization. Whether you sync 3x 1G in parallel or 1x 3G makes > > > no noticeable difference. > > > Speed in general does not increase by increasing the number of sync > > > processes. > > > However, if one want to sync to multiple hosts, the effort may be > > > worthwhile. > > > What is the current advice about speeding up data transfer ? > > > > -- > > Kevin Korb Phone: (407) 252-6853 > > Hi Kevin. What about a single host to another with multiple CPU's ? rsync really needs a lot of CPU power. It is not so much disk-bound (as many conclude against using multiple processes), but cpu-bound. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Speeding up rsync
‐‐‐ Original Message ‐‐‐ On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync wrote: > In my experience backing up multiple hosts to a single host can speed > things up. Especially if using rsync over ssh with multiple CPUs on the > single host. You would need to do some experimentation to determine the > best number for your hardware and network. Also, if you exceed that > number you can put more stress on the backup machine lessening the > impact to the hosts being backed up. > > On 8/24/21 3:58 PM, hancooper via rsync wrote: > > > It makes sense to we to run multiple rsync commands to speed up a backup. > > At work, some have argued that if I sync the data all to the same host, > > there is no advantage > > in parallelization. Whether you sync 3x 1G in parallel or 1x 3G makes > > no noticeable difference. > > Speed in general does not increase by increasing the number of sync > > processes. > > However, if one want to sync to multiple hosts, the effort may be > > worthwhile. > > What is the current advice about speeding up data transfer ? > > -- > Kevin Korb Phone: (407) 252-6853 Hi Kevin. What about a single host to another with multiple CPU's ? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Speeding up rsync
In my experience backing up multiple hosts to a single host can speed things up. Especially if using rsync over ssh with multiple CPUs on the single host. You would need to do some experimentation to determine the best number for your hardware and network. Also, if you exceed that number you can put more stress on the backup machine lessening the impact to the hosts being backed up. On 8/24/21 3:58 PM, hancooper via rsync wrote: > It makes sense to we to run multiple rsync commands to speed up a backup. > > At work, some have argued that if I sync the data all to the same host, > there is no advantage > in parallelization. Whether you sync 3x 1G in parallel or 1x 3G makes > no noticeable difference. > > Speed in general does not increase by increasing the number of sync > processes. > > However, if one want to sync to multiple hosts, the effort may be > worthwhile. > > What is the current advice about speeding up data transfer ? > > > > > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Speeding up rsync
It makes sense to we to run multiple rsync commands to speed up a backup. At work, some have argued that if I sync the data all to the same host, there is no advantage in parallelization. Whether you sync 3x 1G in parallel or 1x 3G makes no noticeable difference. Speed in general does not increase by increasing the number of sync processes. However, if one want to sync to multiple hosts, the effort may be worthwhile. What is the current advice about speeding up data transfer ?-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
