Bob, "Buffering response body" message is written ONLY inside getResponseBody, which in its turn uses getResponseBodyAsStream. See for yourself:
http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/HttpMethodBase.html#681 It means what there's a bit of code in your application that calls getResponseBody or getResponseBodyAsString methods, which causes the buffering of the response body. getResponseBodyAsStream does reconstruct the input stream in case the response has been already buffered, but per default it returns the raw input stream, when available. Feel free to examine the source: http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/HttpMethodBase.html#709 What would HttpClient be worth if it always buffered the response body? I have no idea what led you to believe that "processResponseHeaders" method buffers the response. The method is even called processResponseHeaders not processResponseBody or something Here's what processResponseBody method does: (1) Gets the raw input stream from the connection http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/HttpMethodBase.html#2014 (2) If the chunk-encoding is used, wraps it with ChunkedInputStream. No buffering http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/HttpMethodBase.html#2038 (3) If Content-Length header is used, wraps it with ContentLengthInputStream, which basically ensures that content will not be read past its length. No buffering http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/HttpMethodBase.html#2072 (4) If there's anything wrong with either Transfer-Encoding or Content-Length header, it just leaves the raw input stream alone. No buffering (5) Finally it attaches a AutoCloseInputStream to the resultant input stream, which is intended to close the underlying connection once the entire response is consumed. No buffering. (6) Keeps the resultant input stream. http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/HttpMethodBase.html#1979 (7) Now, if getResponseBodyAsStream is called, it will simply return the input stream without any further manipulations: http://jakarta.apache.org/commons/httpclient/xref/org/apache/commons/httpclient/HttpMethodBase.html#710 (8) End of story Hope this clarifies things a bit Oleg On Wed, 2004-09-29 at 21:17, St Jacques, Robert wrote: > Oleg, > > This is exactly what I tried to do, but the HttpClient seems to buffer the > response anyway. I poked through the code a bit, and did some searching on > the mailing list, and this seems to be the case. Also, when I run tests, > the log output from the HttpClient seems to indicate that the request is > being buffered before I even get an opportunity to call > "getResponseBodyAsStream". I tried this on a request that is about 17 MB > and just sat back and watched as the response was buffered; I ended up > terminating the program after a minute or two without ever actually getting > past the execute method call. Here is the output from the logs: > > enter HttpMethodBase.processResponseHeaders(HttpState, HttpConnection) > enter GetMethod.readResponseBody(HttpState, HttpConnection) > enter HttpMethodBase.readResponseBody(HttpState, HttpConnection) > enter HttpMethodBase.readResponseBody(HttpState, HttpConnection) > enter HttpConnection.getResponseInputStream() > Buffering response body > <lots of logs detailing the bytes read> > > >From here it looks like the "processResponseHeaders" method (which is called > by the HttpMethodBase during an execute) automatically buffers the response. > Is this true, or am I totally misreading the signs? > > Thanks, > Bob > > -----Original Message----- > From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] > Sent: Wednesday, September 29, 2004 3:04 PM > To: Commons HttpClient Project > Subject: Re: streaming responses > > > Bob, > > There's no special magic involved. Make sure you use > HttpMethod#getResponseBodyAsStream, which will return the raw input stream, > and not its buffering counterparts HttpMethod#getResponseBodyAsString and > HttpMethod#getResponseBody. > > You should not worry about chunking. HttpClient will decode chunked input on > fly, if necessary. Just grab the raw input stream and do whatever response > retrieval suits your application best. > > Hope this helps > > Oleg > > > On Wed, 2004-09-29 at 20:43, St Jacques, Robert wrote: > > Howdy, > > > > I've just started using HttpClient for testing. The product that I am > > developing includes a software download feature that downloads > > (sometimes very large) files over the internet using HTTP. The reason > > we use HTTP is that many of our customers are unwilling to open their > > proxies or firewalls to other protocols. > > > > In order to accommodate these downloads, we must either be able to > > stream the response to a GET, or we need to be able to insure that the > > response we get from the server will be chunked in such a way that the > > chunks are small enough to buffer. I know that HttpClient supports > > chunking, but I am unable to 'force' our web server to chunk up the > > data. Furthermore, the HTTP 1.1 RFC indicates that the server does > > not need to guarantee that a response, even if chunked, will be small > > enough to conveniently buffer on the client. > > > > What is the best way to use HttpClient to retrieve large amounts of > > data (as large as 200 MB or more)? > > > > Thanks, > > Bob St. Jacques > > Member Research Technical Staff > > Xerox Corp. > > (585)231-8306 > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
