On Sat, 2013-01-05 at 14:38 -0800, vigna wrote: > Is there any suggestion for parameters of the asynchronous client in case of > several thousands parallel requests (e.g., for the IOReactor)? We are > experimenting both with DefaulHttpClient and DefaultHttpAsyncClient, and > with the same configuration (e.g., 4000 threads using DefaultHttpClient or > 64 threads pushing 4000 async requests into a default > DefaultHttpAsyncClient) we see completely different behaviours. The sync > client fetches more than 10000 pages/s, the async client speed fetches > 50 p/s. >
>From my personal experience a decent blocking HTTP client can be expected to outperform a decent non-blocking HTTP client by 50 to 100%, but such a massive difference does look very suspicious. My guess, though, that the way pages are being processed can be a limiting factor more more than the way they are being retrieved. How do you parse / process the content of the pages? Is your processing code based on standard java InputStream APIs? > Should we increase the number of threads No, you should not. There is no point having more I/O threads than the number of physical CPU cores. > or the I/O interval of the > IOReactor? No, you should not. This will have no impact on performance of what so ever. By reducing the select interval one can get more granular socket timeouts (which is cannot be less than 1 second with the default select interval of 1 second). Oleg --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
