I have a servlet that reads some parameters from a URL, uses an
HttpURLConnection to query and retrieve a web page, parse the text, and finally
construct some XML as output. I know using an HTTP connection as the primary
back-end data source is horribly bad, but we have to do it for now and the web
pages it is scraping are all internal to our large company.
The remote server sometimes takes a long time to respond. The
HttpURLConnection/BufferedReader methods seem to act as if they are
synchronized--only one connection/request can be made at a time. This can
cause significant congestion when multiple requests come in at once.
My goal was to use HttpClient with the MultiThreadedHttpConnectionManager to
handle all of these back-end connections, but so far it seems like the behavior
is the same. Is it possible to have multiple threads from my servlet open
simultaneous connections to an external web site with different requests and
read the results? I would really like to get around the blocking aspect of the
read statements. My servlet declares the MTHCM as a static object in the
servlet class, so that once instance is shared amongst all incoming servlet
requests, and a static{} block is used in the class to set timeouts/etc. The
HttpClient is newed in the doGet method.
If I build a test servlet that does nothing but return a string after about 20
seconds to simulate the remote server responding slowly and point my real
servlet to the test servlet, no other requests/threads to the real servlet are
allowed to make a connection until the test servlet returns it's result. What
am I doing wrong? How can I allow other threads to continue to query the
server, even when one request is very slow to respond? Thank you in advance
for you help!
Josh
____________________________________________________________________________________
Building a website is a piece of cake. Yahoo! Small Business gives you all the
tools to get online.
http://smallbusiness.yahoo.com/webhosting
