Re: threaded rsync to improve performance over long distance ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 rsync does not use scp in any way. On 01/20/13 14:47, Markus Moeller wrote: I try to rsync a lot of data over a long distance AU to UK and get a slow data rate because of the known slow scp performance over long distances. Would it be possible with rsync to run multiple scp process at the same time ? Thank you Markus - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlD8Xm8ACgkQVKC1jlbQAQeZbgCeKeCMPL2wkakiqYu9oKQ2S7Z6 Sq4An0fRzgLhBbg3ZYeaTjcaCo+OSco4 =YNSj -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: threaded rsync to improve performance over long distance ?
At the risk of stating the obvious, is there any way you can break your datasets into smaller pieces? Then, you could start several rsync processes in parallel - one for each piece, possibly using separate Internet connections. As Kevin says, I'm sure there is no direct relationship between rsync and scp - they are two separate transfer protocols, but the above might get around whatever bottleneck you are running into. If rsync is slow, then any other protocol (taking the same path) will probably be just as slow. You can't get a wider pipe by changing faucets. I don't know much about networking, but if you could use network diagnostic tools like ping and traceroute to find the actual cause(s) of the bottleneck (without getting accused of security violations), then you might be able to find a solution. E.g. maybe you could arrange to send your data somewhere closer first with better bandwidth and use their connection (hopefully faster as well - with their permission/service agreement) to send it the rest of the way. Joe On 01/20/2013 02:47 PM, Markus Moeller wrote: I try to rsync a lot of data over a long distance AU to UK and get a slow data rate because of the known slow scp performance over long distances. Would it be possible with rsync to run multiple scp process at the same time ? Thank you Markus -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: threaded rsync to improve performance over long distance ?
Hi Kevin, but rsync uses ssh doesn't it ? Are there any tuning options for buffers to deal with the tcp SYN SYN-ACK delay ? Thank you Markus Kevin Korb k...@sanitarium.net wrote in message news:50fc5e6f.1000...@sanitarium.net... -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 rsync does not use scp in any way. On 01/20/13 14:47, Markus Moeller wrote: I try to rsync a lot of data over a long distance AU to UK and get a slow data rate because of the known slow scp performance over long distances. Would it be possible with rsync to run multiple scp process at the same time ? Thank you Markus - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Florida k...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlD8Xm8ACgkQVKC1jlbQAQeZbgCeKeCMPL2wkakiqYu9oKQ2S7Z6 Sq4An0fRzgLhBbg3ZYeaTjcaCo+OSco4 =YNSj -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: threaded rsync to improve performance over long distance ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You can use the HPN enhancement patches for ssh. They do make a big difference: http://www.psc.edu/index.php/hpn-ssh On 01/20/13 18:22, Markus Moeller wrote: Hi Kevin, but rsync uses ssh doesn't it ? Are there any tuning options for buffers to deal with the tcp SYN SYN-ACK delay ? Thank you Markus Kevin Korb k...@sanitarium.net wrote in message news:50fc5e6f.1000...@sanitarium.net... rsync does not use scp in any way. On 01/20/13 14:47, Markus Moeller wrote: I try to rsync a lot of data over a long distance AU to UK and get a slow data rate because of the known slow scp performance over long distances. Would it be possible with rsync to run multiple scp process at the same time ? Thank you Markus -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlD8fOkACgkQVKC1jlbQAQcqTwCeMsxo3MMyfFx3omGpIu5ELDuM V64AoLdq5ak55CPIIjVk7AHK6U+0XTbN =pu+/ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: threaded rsync to improve performance over long distance ?
With the SSH-HPN you can achieve over 200MB/sec. Depends on your latency and packet drops but you should certainly try a modified ssh/scp with rsync. Also are you using compression with rsync -z? That's an area that could be multi threaded. On Jan 20, 2013, at 3:25 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You can use the HPN enhancement patches for ssh. They do make a big difference: http://www.psc.edu/index.php/hpn-ssh On 01/20/13 18:22, Markus Moeller wrote: Hi Kevin, but rsync uses ssh doesn't it ? Are there any tuning options for buffers to deal with the tcp SYN SYN-ACK delay ? Thank you Markus Kevin Korb k...@sanitarium.net wrote in message news:50fc5e6f.1000...@sanitarium.net... rsync does not use scp in any way. On 01/20/13 14:47, Markus Moeller wrote: I try to rsync a lot of data over a long distance AU to UK and get a slow data rate because of the known slow scp performance over long distances. Would it be possible with rsync to run multiple scp process at the same time ? Thank you Markus -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin KorbPhone:(407) 252-6853 Systems AdministratorInternet: FutureQuest, Inc.ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page:http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlD8fOkACgkQVKC1jlbQAQcqTwCeMsxo3MMyfFx3omGpIu5ELDuM V64AoLdq5ak55CPIIjVk7AHK6U+0XTbN =pu+/ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
