Re: [lftp] lftp-4.4.13 -- multi-core/multi-threading support for get on 10GbE networks?
-Original Message- From: Alexander V. Lukyanov [mailto:l...@netis.ru] Sent: Friday, November 29, 2013 2:23 AM To: Justin Piszcz Cc: lftp@uniyar.ac.ru Subject: Re: [lftp] lftp-4.4.13 -- multi-core/multi-threading support for get on 10GbE networks? On Thu, Nov 28, 2013 at 03:47:54PM -0500, Justin Piszcz wrote: When transferring data on high speed networks (10GbE) lftp hits 100% on a moderately fast Xeon CPU (E5645), the FTP server is not the bottleneck as it uses around 37% CPU (different CPU on the server host). Are there any plans to spin off separate workers (if possible) so a single CPU-core is not a bottleneck at the client-side? I don't think multithreading is going to be implemented in lftp. I avoided it from the start as single-threading makes programming and debugging easier. But I think it is possible to squeeze more performance by optimization. First provide me with profiling information (compile with -pg gcc option, then run lftp, then run gprof, send me the output), then be ready to try optimized versions to see if they make a difference. -- Alexander. Hello, I forgot I had -debug enabled from my earlier testing when we were tracking down that cls bug, when debug is disabled, lftp is nearly as fast as NFS-- so I think performance is good for now. If further tuning/gprof is needed I can run through it if necessary but I'm happy with the speeds now. Device eth4 [192.168.1.2] (1/1): Incoming: Outgoing: Curr: 0.81 MByte/s Curr: 841.87 MByte/s Avg: 0.76 MByte/s Avg: 800.37 MByte/s Min: 0.58 MByte/s Min: 602.02 MByte/s Max: 0.81 MByte/s Max: 841.87 MByte/s Ttl: 1.32 GByte Ttl: 203.12 GByte Justin. ___ lftp mailing list lftp@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp
Re: [lftp] lftp-4.4.13 -- multi-core/multi-threading support for get on 10GbE networks?
On Thu, Nov 28, 2013 at 03:47:54PM -0500, Justin Piszcz wrote: When transferring data on high speed networks (10GbE) lftp hits 100% on a moderately fast Xeon CPU (E5645), the FTP server is not the bottleneck as it uses around 37% CPU (different CPU on the server host). Are there any plans to spin off separate workers (if possible) so a single CPU-core is not a bottleneck at the client-side? I don't think multithreading is going to be implemented in lftp. I avoided it from the start as single-threading makes programming and debugging easier. But I think it is possible to squeeze more performance by optimization. First provide me with profiling information (compile with -pg gcc option, then run lftp, then run gprof, send me the output), then be ready to try optimized versions to see if they make a difference. -- Alexander. ___ lftp mailing list lftp@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp
[lftp] lftp 4.4.13
lftp-4.4.13 has been released. Changes: * new option -l (--ls) for find command. * improve workaround for single NL replies from an FTP server. * Ukrainian translation updated (Yuri Chornoivan). * fixed spinning in get when no remote session is open. * don't pre-fetch file information in get when not needed. * fixed handling of 400/501 http codes for PROPFIND to switch to HEAD. * fixed a crash after cls. * added file size decrease checking. * used a newer libtool for ppc64le platform. Get it from http://lftp.yar.ru/get.html or your favorite mirror. (4.4.12 was an unfortunate release with a last minute bug and was withdrawn). -- Alexander. ___ lftp mailing list lftp@uniyar.ac.ru http://univ.uniyar.ac.ru/mailman/listinfo/lftp
