Hi! Well, 100 Mb/sec on a single core sounds about right. I have seen about 4 GBit/sec uploads over multiple connections (cores!), which corresponds to your 500 Mb/sec for multiple connections. I never considered that slow before ;-).
urlencoding is not really meant for large amounts of data (why would anyone even DO that!?), so the significantly lower performance is hardly surprising. You might be able to write a faster, custom implementation of strtoul for the 2-byte hexadecimal numbers that are being decoded and speed it up by 2x, but decoding this will never be fast. Speeding up the process_value_to_boundary memcmp will be virtually impossible. Here, what might be possible is to use "PUT" (from JavaScript) or otherwise convince the browser to use a more server-friendly upload encoding --- mostly by not using an HTML form to generate the upload. If you use such a custom encoding (ideally: no encoding, just raw binary data!), you would then not even initialize the MHD_PostProcessor and instead directly process the received "upload_data" yourself. Happy hacking! Christian On 10/27/13 02:27, Adam Homolya wrote: > Hello, > > I did some profiling with callgrind today after raising the buffer size to > 1024 Kbyte with my application. It turned out if post data is urlencoded, > libmicrohttpd can do not more than ~10 Mb/sec on a new gen i7 machine, and > the limit is the cpu. The application spends 76% of its time within > MHD_http_unescape, executing strtoul (67%). > In case of multipart data, the performance is better, can perform up to 100 > Mb/sec while uploading with one client (~500 Mb/s for multiple connections). > The limit is the cpu here too. The profiling showed the application spends > 97% of its time in process_value_to_boundary, doing memcmp. > > Kind regards, > Adam > > -----Original Message----- > From: Christian Grothoff [mailto:[email protected]] > Sent: 2013. október 25. 19:54 > To: Adam Homolya > Cc: Peter Saghelyi; Attila Body; Gabor Mihaly > Subject: Re: libmicrohttpd blocking response from callback > > On 10/25/2013 07:47 PM, Adam Homolya wrote: >> Dear Christian, >> >> Thank you for your fast response, the solution is excellent. >> I have one more problem though. >> I've been benchmarking libmicrohttpd recently, got really good results for >> reading large files, using response form callback. However posting large >> files seem to give rather bad results, and I would like to ask if I miss >> something. I used post_example.c for the file upload method, using 64K post >> buffer size. When uploading a large file with a single client on localhost >> the system monitor shows one of the CPU cores are always on 100% usage (if I >> raise uploading processes to 8, all 8 cores are on 100%), and the upload >> speed is always around only 9-10Mbyte / sec which seems a bit poor for me >> (download speed can reach 4.5 Gbyte/sec when redirecting download result to >> /dev/null immediately). I don't process the post data in my application's >> logic, so that can't be the bottleneck. Can I boost somehow this >> performance? Thank you very much for your help. > > I don't know how your application is encoding the POST or how your > application logic is handling it. > 100% CPU at 9-10 MByte is certainly, eh, unusual for a modern machine. > I've seen upload speeds at > 50% of download speeds (that's normal because for uploads we don't support > recvfile and we have to possibly do parsing), but your case seems to be much > worse. I'd check > 1) if changing the encoding by the client for the POST data helps > 2) if changing the buffer size (64k is not much) helps --- if you have the > RAM try 512 KB and > see if it does anything > 3) run a profiler to see where the CPU is going to. > > Happy hacking! > > Christian > > >> Kind regards >> Adam >> >> >> -----Original Message----- >> From: Christian Grothoff [mailto:[email protected]] >> Sent: 2013. október 24. 17:39 >> To: Adam Homolya >> Cc: Attila Body; Peter Saghelyi >> Subject: Re: libmicrohttpd blocking response from callback >> >> Dear Adam, >> >> This was discussed on the mailinglist today and a solution was committed to >> SVN HEAD today as well. Note that the documentation on the website is thus >> outdated with respect to this change; however, if you rebuild the >> documentation from SVN HEAD and look for "resume" and "suspend" you should >> find details for how to do this. >> >> With the new code, it is possible to return from the callback and not create >> busy waiting. >> >> I hope this helps! >> >> >> Best regards, >> >> Christian >> >> On 10/24/2013 04:02 PM, Adam Homolya wrote: >>> Dear Christian, >>> >>> First of all sorry for disturbing you. At my company, we are planning to >>> use libmicrohttpd as a web server frontend for one of our projects. I've >>> been evaluating the library for a while and I encountered a serious problem >>> (didn't find any solution in the docs). >>> I would like to stream large amount of data for the clients. The obvious >>> solution for that would be creating response from callback >>> (MHD_create_response_from_callback). My problem is that libmicrohttpd >>> invokes the content reader callback when it wants to send data, and not >>> when data is available, so that it could block, which is not an option for >>> us. Returning from the callback (marking that no data was read) causes >>> "busy wait" like behavior which is even worse. I am looking for some way >>> for telling the library not to call the content reader callback until data >>> is available for read in a non-blocking manner. Is it possible to work out >>> somehow? Thank you very much. >>> >>> Best Regards, >>> >>> -- >>> Adam Almos Homolya >>> Software Engineer >>> LogMeIn Inc. >>>
